2026-03-07 02:35:48,560 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-03-07 02:35:48,569 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-03-07 02:35:48,575 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-03-07 02:35:48,576 INFO [rotate_logs] log files rotated
2026-03-07 02:35:48,589 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-03-07 02:38:24,596 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/06mar26
2026-03-07 02:38:24,616 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-06 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/06mar26
2026-03-07 02:38:24,618 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-06 forecast2 completed
2026-03-07 02:38:25,055 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-07 02:40:34,673 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 387, 71.7% complete
2026-03-07 02:45:34,674 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 480, 88.9% complete
2026-03-07 02:50:34,675 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-07 02:50:34,679 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-07 02:50:35,056 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 02:50:36,478 INFO [download_wwatch3_results] running in process 1151112
2026-03-07 02:50:36,479 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 02:50:36,480 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-07 02:50:36,481 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-07 02:51:12,486 INFO [download_wwatch3_results] forecast2 2026-03-07 results files from arbutus.cloud-nowcast downloaded
2026-03-07 02:51:12,490 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-07 02:51:12,636 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 02:51:13,834 INFO [update_forecast_datasets] running in process 1151121
2026-03-07 02:51:13,835 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 02:51:13,837 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-07 02:51:13,838 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-07 02:51:13,839 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/07mar26 run
2026-03-07 02:51:13,867 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/07mar26 run
2026-03-07 02:51:13,868 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/07mar26 run
2026-03-07 02:51:45,096 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/07mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-07 02:51:45,097 INFO [update_forecast_datasets] wwatch3 2026-03-07 forecast2 rolling forecast datasets updated
2026-03-07 02:51:45,101 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-07 02:51:45,273 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-07 02:51:46,542 INFO [ping_erddap] running in process 1151161
2026-03-07 02:51:46,543 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 02:51:46,544 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-07 02:51:46,546 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-07 02:51:46,548 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-07 02:51:46,550 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-07 02:51:46,672 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 02:51:50,327 INFO [make_plots] running in process 1151166
2026-03-07 02:51:50,329 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 02:51:50,330 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-07 02:51:50,330 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-07 02:51:50,331 INFO [make_plots] preparing render list for 2026-03-07 WaveWatch3 forecast2 publish figures
2026-03-07 02:51:50,332 INFO [make_plots] added Halibut Bank figure to 2026-03-07 WaveWatch3 forecast2 publish render list
2026-03-07 02:51:50,333 INFO [make_plots] added Sentry Shoal figure to 2026-03-07 WaveWatch3 forecast2 publish render list
2026-03-07 02:51:50,334 INFO [make_plots] starting to render wwatch3 forecast2 publish 07mar26 figures
2026-03-07 02:51:50,953 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-07 02:51:57,157 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-07 02:52:01,426 INFO [make_plots] finished rendering wwatch3 forecast2 publish 07mar26 figures
2026-03-07 02:52:01,427 INFO [make_plots] wwatch3 publish plots for 2026-03-07 forecast2 completed
2026-03-07 02:52:01,431 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-07 07:42:49,584 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260307/12/
2026-03-07 07:42:49,586 INFO [collect_weather] 2026-03-07 2.5km weather forecast 12 collection complete
2026-03-07 07:42:49,590 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-07 07:42:49,961 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-03-06'], host='localhost')
2026-03-07 07:42:49,968 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-03-06'], host='localhost')
2026-03-07 07:42:49,972 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-03-06'], host='localhost')
2026-03-07 07:42:49,977 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-03-06'], host='localhost')
2026-03-07 07:42:49,985 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-03-07 07:42:49,990 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-03-07 07:42:49,994 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-03-07 07:42:49,999 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-03-07 07:42:50,011 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-03-07 07:42:50,476 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260307/12/
2026-03-07 07:42:50,477 INFO [crop_gribs] 2026-03-07 12 GRIBs cropping complete
2026-03-07 07:42:50,479 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-07 07:42:51,077 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-03-07 07:42:51,702 INFO [collect_weather] running in process 1154957
2026-03-07 07:42:51,703 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:51,704 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-07 07:42:51,706 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:51,707 INFO [collect_NeahBay_ssh] running in process 1154955
2026-03-07 07:42:51,708 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:51,708 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-03-07 07:42:51,709 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:51,710 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-03-07
2026-03-07 07:42:51,728 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-03-07 07:42:51,839 INFO [download_live_ocean] running in process 1154956
2026-03-07 07:42:51,839 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:51,840 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2026-03-07 07:42:51,841 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:51,842 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-03-07
2026-03-07 07:42:52,253 INFO [collect_river_data] running in process 1154952
2026-03-07 07:42:52,254 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:52,255 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-07 07:42:52,256 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:52,257 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-03-06
2026-03-07 07:42:52,266 INFO [collect_river_data] running in process 1154949
2026-03-07 07:42:52,267 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:52,268 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-07 07:42:52,269 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:52,270 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-03-06
2026-03-07 07:42:52,284 INFO [collect_river_data] running in process 1154951
2026-03-07 07:42:52,285 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:52,285 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-07 07:42:52,286 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:52,287 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-03-06
2026-03-07 07:42:52,291 INFO [collect_river_data] running in process 1154953
2026-03-07 07:42:52,292 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:52,292 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-07 07:42:52,293 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:52,295 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-03-06
2026-03-07 07:42:52,391 INFO [make_turbidity_file] running in process 1154954
2026-03-07 07:42:52,392 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:52,393 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-03-07 07:42:52,394 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:52,395 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-03-07
2026-03-07 07:42:52,552 INFO [download_live_ocean] downloaded 10198682 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.03.07/ubc.nc to /results/forcing/LiveOcean/downloaded/20260307/low_passed_UBC.nc
2026-03-07 07:42:52,553 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-03-07 07:42:52,718 INFO [crop_gribs] running in process 1154958
2026-03-07 07:42:52,719 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:52,720 INFO [crop_gribs] publishing log messages to tcp://*:5570
2026-03-07 07:42:52,721 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:52,722 INFO [crop_gribs] cropping 2026-03-07 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-03-07 07:42:52,737 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260307/18/
2026-03-07 07:42:52,797 INFO [collect_NeahBay_ssh] 2026-03-07T00:00:00-08:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-03-07 07:42:52,800 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-07 07:42:52,853 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-03-06&endDT=2026-03-06 "HTTP/1.1 200 "
2026-03-07 07:42:52,858 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-03-06 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-03-07 07:42:52,858 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-03-06 completed
2026-03-07 07:42:52,861 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-03-06&endDT=2026-03-06 "HTTP/1.1 200 "
2026-03-07 07:42:52,861 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-03-06&endDT=2026-03-06 "HTTP/1.1 200 "
2026-03-07 07:42:52,862 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-03-06&endDT=2026-03-06 "HTTP/1.1 200 "
2026-03-07 07:42:52,872 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-03-06 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-03-07 07:42:52,872 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-03-06 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-03-07 07:42:52,873 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-03-06 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-03-07 07:42:52,873 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-03-06 completed
2026-03-07 07:42:52,874 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-03-06 completed
2026-03-07 07:42:52,874 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-03-06 completed
2026-03-07 07:42:53,026 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 07:42:53,031 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-07 07:42:53,203 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-07 07:42:53,211 INFO [grib_to_netcdf] running in process 1155038
2026-03-07 07:42:53,212 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:53,213 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2026-03-07 07:42:53,214 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:53,215 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-07 nowcast and forecast runs
2026-03-07 07:42:53,421 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-07 07:42:53,568 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260307/low_passed_UBC.nc
2026-03-07 07:42:53,569 INFO [download_live_ocean] 2026-03-07 Live Ocean file for Salish Sea western boundary downloaded
2026-03-07 07:42:53,635 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-07 07:42:53,796 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-03-07 07:42:54,181 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-03-07'], host='localhost')
2026-03-07 07:42:55,093 INFO [make_turbidity_file] 2026-03-07 Fraser River turbidity file creation complete
2026-03-07 07:42:55,095 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-03-07 07:42:56,032 INFO [make_ssh_files] running in process 1155092
2026-03-07 07:42:56,033 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:56,034 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-03-07 07:42:56,035 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:56,036 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-03-07 NOAA Neah Bay 06Z observation and forecast values
2026-03-07 07:42:56,473 INFO [make_live_ocean_files] running in process 1155112
2026-03-07 07:42:56,474 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:42:56,475 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2026-03-07 07:42:56,476 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:42:56,477 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-03-07 Live Ocean run
2026-03-07 07:42:57,743 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d03.nc
2026-03-07 07:42:57,928 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d04.nc
2026-03-07 07:42:58,120 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d05.nc
2026-03-07 07:42:58,296 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d06.nc
2026-03-07 07:42:58,502 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d07.nc
2026-03-07 07:42:58,725 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d08.nc
2026-03-07 07:42:58,932 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d09.nc
2026-03-07 07:42:59,120 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d10.nc
2026-03-07 07:42:59,253 INFO [make_ssh_files] sea surface height boundary file for 2026-03-07 nowcast run created
2026-03-07 07:42:59,257 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-07 07:42:59,409 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-07 07:42:59,415 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-07 07:43:00,786 INFO [make_201702_runoff_file] running in process 1155175
2026-03-07 07:43:00,787 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:00,789 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2026-03-07 07:43:00,789 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:00,827 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-07 07:43:00,956 INFO [make_201702_runoff_file] 2026-03-07 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-07 07:43:00,959 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-07 07:43:01,501 INFO [make_runoff_file] running in process 1155173
2026-03-07 07:43:01,502 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:01,503 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2026-03-07 07:43:01,504 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:01,505 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-06
2026-03-07 07:43:04,094 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-06: /results/forcing/rivers/R202108Dailies_y2026m03d06.nc
2026-03-07 07:43:04,095 INFO [make_runoff_file] 2026-03-06 runoff file creation completed
2026-03-07 07:43:04,097 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-07 07:43:06,045 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m03d07.nc
2026-03-07 07:43:15,046 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d08.nc
2026-03-07 07:43:20,314 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d09.nc
2026-03-07 07:43:20,329 INFO [grib_to_netcdf] 2026-03-07 NEMO-atmos forcing file for nowcast+ created
2026-03-07 07:43:20,331 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-07 07:43:37,568 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d07.nc
2026-03-07 07:43:37,569 INFO [make_live_ocean_files] 2026-03-07 Live Ocean western boundary conditions files created
2026-03-07 07:43:37,571 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-03-07 07:43:37,723 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-03-07 07:43:37,726 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-03-07 07:43:37,730 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-03-07 07:43:37,734 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-03-07 07:43:37,741 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-03-07 07:43:38,968 INFO [ping_erddap] running in process 1155252
2026-03-07 07:43:38,969 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:38,970 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-07 07:43:38,972 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:38,973 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-03-07 07:43:38,975 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-07 07:43:38,991 INFO [upload_forcing] running in process 1155257
2026-03-07 07:43:38,992 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:38,993 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-07 07:43:38,995 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:39,038 INFO [upload_forcing] running in process 1155254
2026-03-07 07:43:39,039 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:39,040 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-07 07:43:39,041 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:39,065 INFO [upload_forcing] running in process 1155256
2026-03-07 07:43:39,066 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:39,067 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-07 07:43:39,068 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:39,078 INFO [upload_forcing] running in process 1155255
2026-03-07 07:43:39,080 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:39,081 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-07 07:43:39,082 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:40,802 INFO [upload_forcing] nowcast+ 2026-03-07 forcing files upload to optimum-hindcast completed
2026-03-07 07:43:40,806 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-07 07:43:43,301 INFO [upload_forcing] nowcast+ 2026-03-07 forcing files upload to arbutus.cloud-nowcast completed
2026-03-07 07:43:43,305 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-07 07:43:43,453 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 07:43:44,818 INFO [make_forcing_links] running in process 1155288
2026-03-07 07:43:44,820 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:44,821 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-03-07 07:43:44,822 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:45,957 INFO [make_forcing_links] nowcast+ 2026-03-07 forcing file links on arbutus.cloud-nowcast created
2026-03-07 07:43:45,973 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-07 07:43:46,121 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-07'], host='arbutus.cloud-nowcast')
2026-03-07 07:43:46,636 INFO [upload_forcing] nowcast+ 2026-03-07 forcing files upload to orcinus-nowcast-agrif completed
2026-03-07 07:43:46,640 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-07 07:43:48,206 INFO [run_NEMO] running in process 19934
2026-03-07 07:43:48,208 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:48,209 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-07 07:43:48,268 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:48,940 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-07 07:43:49,124 INFO [upload_forcing] nowcast+ 2026-03-07 forcing files upload to robot.nibi completed
2026-03-07 07:43:49,129 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-07 07:43:49,248 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/07mar26nowcast_2026-03-07T074348.256543-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-07 07:43:49,262 INFO [run_NEMO] nowcast NEMO run for 2026-03-07 on arbutus.cloud-nowcast started
2026-03-07 07:43:49,282 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-07 07:43:49,430 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-07 07:43:51,442 INFO [watch_NEMO] running in process 20207
2026-03-07 07:43:51,443 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 07:43:51,444 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-07 07:43:51,445 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-07 07:43:51,538 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-07 07:48:51,539 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19081911 = 2026-03-07 05:13:20 UTC, 21.8% complete
2026-03-07 07:53:51,540 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19082435 = 2026-03-07 11:02:40 UTC, 46.0% complete
2026-03-07 07:58:51,541 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19082957 = 2026-03-07 16:50:40 UTC, 70.2% complete
2026-03-07 08:03:51,543 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19083473 = 2026-03-07 22:34:40 UTC, 94.1% complete
2026-03-07 08:08:51,547 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-03-07 08:08:51,552 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-07 08:08:51,933 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:08:51,941 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:08:53,310 INFO [download_results] running in process 1155545
2026-03-07 08:08:53,311 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:08:53,313 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-07 08:08:53,314 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:08:53,315 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/07mar26 to /results/SalishSea/nowcast-blue.202111
2026-03-07 08:08:53,437 INFO [make_forcing_links] running in process 1155544
2026-03-07 08:08:53,438 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:08:53,439 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-07 08:08:53,441 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:08:54,443 INFO [make_forcing_links] ssh 2026-03-07 forcing file links on arbutus.cloud-nowcast created
2026-03-07 08:08:54,453 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-07 08:08:54,629 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-07'], host='arbutus.cloud-nowcast')
2026-03-07 08:08:56,812 INFO [run_NEMO] running in process 20925
2026-03-07 08:08:56,813 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:08:56,815 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-07 08:08:56,816 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:08:57,599 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-07 08:08:57,918 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/07mar26forecast_2026-03-07T080856.881616-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-07 08:08:57,931 INFO [run_NEMO] forecast NEMO run for 2026-03-07 on arbutus.cloud-nowcast started
2026-03-07 08:08:57,936 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-07 08:08:58,104 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-07 08:09:00,108 INFO [watch_NEMO] running in process 21198
2026-03-07 08:09:00,109 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:09:00,110 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-07 08:09:00,112 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:09:00,175 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-07 08:09:23,219 INFO [download_results] nowcast 2026-03-07 results files from arbutus.cloud-nowcast downloaded
2026-03-07 08:09:23,223 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-07 08:09:23,385 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:09:23,390 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-03-06'], host='localhost')
2026-03-07 08:09:23,395 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:09:25,354 INFO [make_CHS_currents_file] running in process 1155566
2026-03-07 08:09:25,355 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:09:25,356 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-07 08:09:25,357 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:09:26,687 INFO [make_plots] running in process 1155563
2026-03-07 08:09:26,708 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:09:26,726 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-07 08:09:26,742 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:09:26,743 INFO [make_plots] running in process 1155565
2026-03-07 08:09:26,744 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:09:26,745 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-07 08:09:26,746 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:09:27,112 INFO [make_plots] preparing render list for 2026-03-07 NEMO nowcast-blue research figures
2026-03-07 08:09:27,132 INFO [make_plots] preparing render list for 2026-03-06 NEMO nowcast-blue comparison figures
2026-03-07 08:09:27,195 INFO [make_plots] added SH_wind figure to 2026-03-06 NEMO nowcast-blue comparison render list
2026-03-07 08:09:27,196 INFO [make_plots] added Compare_VENUS_East figure to 2026-03-06 NEMO nowcast-blue comparison render list
2026-03-07 08:09:27,197 INFO [make_plots] added Compare_VENUS_Central figure to 2026-03-06 NEMO nowcast-blue comparison render list
2026-03-07 08:09:27,198 INFO [make_plots] starting to render nemo nowcast comparison 06mar26 figures
2026-03-07 08:09:29,138 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-blue research render list
2026-03-07 08:09:29,849 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-blue research render list
2026-03-07 08:09:29,850 INFO [make_plots] added Currents_sections_and_surface figure to 2026-03-07 NEMO nowcast-blue research render list
2026-03-07 08:09:29,851 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-03-07 NEMO nowcast-blue research render list
2026-03-07 08:09:29,852 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-03-07 NEMO nowcast-blue research render list
2026-03-07 08:09:29,853 INFO [make_plots] starting to render nemo nowcast research 07mar26 figures
2026-03-07 08:09:59,388 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-07 for nowcast
2026-03-07 08:09:59,391 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-07 08:10:38,204 INFO [make_plots] finished rendering nemo nowcast research 07mar26 figures
2026-03-07 08:10:38,206 INFO [make_plots] nemo research plots for 2026-03-07 nowcast completed
2026-03-07 08:10:38,223 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-07 08:11:10,269 INFO [make_plots] finished rendering nemo nowcast comparison 06mar26 figures
2026-03-07 08:11:10,272 INFO [make_plots] nemo comparison plots for 2026-03-06 nowcast completed
2026-03-07 08:11:10,276 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-07 08:14:00,174 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19084091 = 2026-03-08 05:26:40 UTC, 15.1% complete
2026-03-07 08:19:00,175 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19084615 = 2026-03-08 11:16:00 UTC, 31.3% complete
2026-03-07 08:24:00,176 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19085144 = 2026-03-08 17:08:40 UTC, 47.6% complete
2026-03-07 08:29:00,177 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19085689 = 2026-03-08 23:12:00 UTC, 64.5% complete
2026-03-07 08:34:00,178 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19086244 = 2026-03-09 05:22:00 UTC, 81.6% complete
2026-03-07 08:39:00,181 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19086779 = 2026-03-09 11:18:40 UTC, 98.1% complete
2026-03-07 08:44:00,185 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-03-07 08:44:00,189 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-07 08:44:00,523 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-07'], host='arbutus.cloud-nowcast')
2026-03-07 08:44:00,535 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-07'], host='arbutus.cloud-nowcast')
2026-03-07 08:44:00,539 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:44:00,546 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:44:00,550 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:44:00,556 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:44:00,560 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:44:02,495 INFO [download_results] running in process 1156311
2026-03-07 08:44:02,496 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:02,496 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-07 08:44:02,497 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:02,498 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/07mar26 to /results/SalishSea/forecast.202111
2026-03-07 08:44:02,631 INFO [upload_forcing] running in process 1156309
2026-03-07 08:44:02,632 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:02,633 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-07 08:44:02,634 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:02,635 INFO [upload_forcing] running in process 1156307
2026-03-07 08:44:02,636 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:02,637 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-07 08:44:02,638 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:02,641 INFO [upload_forcing] running in process 1156308
2026-03-07 08:44:02,642 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:02,643 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-07 08:44:02,644 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:02,661 INFO [upload_forcing] running in process 1156310
2026-03-07 08:44:02,663 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:02,663 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-07 08:44:02,664 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:03,396 INFO [upload_forcing] turbidity 2026-03-07 forcing files upload to optimum-hindcast completed
2026-03-07 08:44:03,399 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-07 08:44:03,505 INFO [make_ww3_wind_file] running in process 22052
2026-03-07 08:44:03,509 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:03,513 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-07 08:44:03,517 INFO [make_ww3_current_file] running in process 22053
2026-03-07 08:44:03,522 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:03,528 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-07 08:44:03,575 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:03,585 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:03,603 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-07 forecast run
2026-03-07 08:44:03,617 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-07 forecast run
2026-03-07 08:44:03,855 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-07 08:44:03,859 INFO [upload_forcing] turbidity 2026-03-07 forcing files upload to arbutus.cloud-nowcast completed
2026-03-07 08:44:04,014 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:44:05,025 INFO [upload_forcing] turbidity 2026-03-07 forcing files upload to orcinus-nowcast-agrif completed
2026-03-07 08:44:05,040 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-07 08:44:07,220 INFO [upload_forcing] turbidity 2026-03-07 forcing files upload to robot.nibi completed
2026-03-07 08:44:07,223 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-07 08:44:08,320 INFO [make_forcing_links] running in process 1156341
2026-03-07 08:44:08,325 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:08,329 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-07 08:44:08,333 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:10,881 INFO [make_forcing_links] nowcast-green 2026-03-07 forcing file links on arbutus.cloud-nowcast created
2026-03-07 08:44:10,885 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-07 08:44:11,064 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-07'], host='arbutus.cloud-nowcast')
2026-03-07 08:44:14,943 INFO [run_NEMO] running in process 22690
2026-03-07 08:44:14,955 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:14,997 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-07 08:44:15,022 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:15,728 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-07 08:44:16,088 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/07mar26nowcast-green_2026-03-07T084414.992292-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-07 08:44:16,143 INFO [run_NEMO] nowcast-green NEMO run for 2026-03-07 on arbutus.cloud-nowcast started
2026-03-07 08:44:16,176 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-07 08:44:16,282 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-07 forecast run
2026-03-07 08:44:16,312 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-03-07 08:44:16,321 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-07 08:44:18,433 INFO [watch_NEMO] running in process 22968
2026-03-07 08:44:18,438 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:18,439 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-07 08:44:18,440 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:44:18,505 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-07 08:44:43,905 INFO [download_results] forecast 2026-03-07 results files from arbutus.cloud-nowcast downloaded
2026-03-07 08:44:43,909 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-07 08:44:44,079 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:44:47,824 INFO [make_CHS_currents_file] running in process 1156358
2026-03-07 08:44:47,826 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:44:47,828 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-07 08:44:47,829 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:45:45,846 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-07 for forecast
2026-03-07 08:45:45,849 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-07 08:45:45,997 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:45:47,752 INFO [update_forecast_datasets] running in process 1156395
2026-03-07 08:45:47,754 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:45:47,755 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-07 08:45:47,757 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:45:47,758 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/07mar26 run
2026-03-07 08:45:48,302 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/07mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-07 08:45:48,303 INFO [update_forecast_datasets] nemo 2026-03-07 forecast rolling forecast datasets updated
2026-03-07 08:45:48,305 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-07 08:45:48,467 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-07 08:45:48,472 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:45:48,478 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 08:45:49,782 INFO [ping_erddap] running in process 1156402
2026-03-07 08:45:49,783 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:45:49,784 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-07 08:45:49,785 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:45:49,796 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-07 08:45:49,797 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-07 08:45:51,937 INFO [make_surface_current_tiles] running in process 1156405
2026-03-07 08:45:51,939 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:45:51,941 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-07 08:45:51,942 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:45:52,019 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-07 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/07mar26
2026-03-07 08:45:54,479 INFO [make_plots] running in process 1156404
2026-03-07 08:45:54,480 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:45:54,481 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-07 08:45:54,482 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:45:54,958 INFO [make_plots] preparing render list for 2026-03-07 NEMO forecast publish figures
2026-03-07 08:45:55,400 INFO [make_plots] added Website_thumbnail figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,401 INFO [make_plots] added Threshold_website figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,402 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,403 INFO [make_plots] added Boundary Bay figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,404 INFO [make_plots] added Campbell River figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,405 INFO [make_plots] added Cherry Point figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,406 INFO [make_plots] added Friday Harbor figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,407 INFO [make_plots] added Halfmoon Bay figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,408 INFO [make_plots] added Nanaimo figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,409 INFO [make_plots] added Neah Bay figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,410 INFO [make_plots] added New Westminster figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,411 INFO [make_plots] added Patricia Bay figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,411 INFO [make_plots] added Point Atkinson figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,413 INFO [make_plots] added Port Renfrew figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,414 INFO [make_plots] added Sand Heads figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,415 INFO [make_plots] added Sandy Cove figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,416 INFO [make_plots] added Squamish figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,417 INFO [make_plots] added Victoria figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,419 INFO [make_plots] added Woodwards Landing figure to 2026-03-07 NEMO forecast publish render list
2026-03-07 08:45:55,420 INFO [make_plots] starting to render nemo forecast publish 07mar26 figures
2026-03-07 08:46:42,797 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-07 forecast run
2026-03-07 08:46:42,801 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-07 08:46:43,109 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-07'], host='arbutus.cloud-nowcast')
2026-03-07 08:46:45,370 INFO [run_ww3] running in process 23091
2026-03-07 08:46:45,371 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:46:45,372 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-07 08:46:45,373 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:46:45,377 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-07T084645.369489-0800
2026-03-07 08:46:45,378 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-07T084645.369489-0800/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-07 08:46:45,390 INFO [run_ww3] nowcast WaveWatch3 run for 2026-03-07 on arbutus.cloud-nowcast started
2026-03-07 08:46:45,394 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-07 08:46:45,704 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-07 08:46:47,718 INFO [watch_ww3] running in process 23178
2026-03-07 08:46:47,719 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 08:46:47,720 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-07 08:46:47,721 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-07 08:46:47,752 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-07 08:49:18,507 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19081612 = 2026-03-07 01:54:00 UTC, 7.9% complete
2026-03-07 08:51:47,752 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2026-03-07 08:54:18,507 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19081803 = 2026-03-07 04:01:20 UTC, 16.8% complete
2026-03-07 08:55:54,833 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-07 08:55:54,834 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-07 08:56:21,624 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 08:56:47,752 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete
2026-03-07 08:57:20,567 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/07mar26
2026-03-07 08:57:29,808 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-07 08:57:29,809 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-07 08:57:48,412 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 08:58:26,045 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 08:58:45,479 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 08:59:05,116 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 08:59:18,507 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19081995 = 2026-03-07 06:09:20 UTC, 25.7% complete
2026-03-07 08:59:24,519 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 08:59:42,674 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 09:00:02,807 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 09:00:21,941 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-07 09:00:21,942 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-07 09:00:41,014 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 09:01:04,099 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-02 00:00:00Z to 2026-03-09 11:55:00Z
2026-03-07 09:01:23,572 INFO [make_plots] finished rendering nemo forecast publish 07mar26 figures
2026-03-07 09:01:23,574 INFO [make_plots] nemo publish plots for 2026-03-07 forecast completed
2026-03-07 09:01:23,581 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-07 09:01:23,742 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:01:29,042 INFO [make_feeds] running in process 1161108
2026-03-07 09:01:29,043 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:01:29,044 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-07 09:01:29,045 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:01:44,944 INFO [make_feeds] ATOM feeds for 2026-03-07 forecast run completed
2026-03-07 09:01:44,947 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-07 09:01:47,753 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 300, 69.4% complete
2026-03-07 09:02:37,289 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/07mar26
2026-03-07 09:02:37,314 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-07 forecast into /results/nowcast-sys/figures/surface_currents/forecast/07mar26
2026-03-07 09:02:37,317 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-07 forecast completed
2026-03-07 09:02:37,661 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-07 09:04:18,508 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19082188 = 2026-03-07 08:18:00 UTC, 34.6% complete
2026-03-07 09:06:47,755 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 399, 92.4% complete
2026-03-07 09:09:18,509 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19082380 = 2026-03-07 10:26:00 UTC, 43.5% complete
2026-03-07 09:11:47,755 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-07 09:11:47,757 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-07 09:11:48,311 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:11:48,318 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-07'], host='arbutus.cloud-nowcast')
2026-03-07 09:11:49,611 INFO [download_wwatch3_results] running in process 1161263
2026-03-07 09:11:49,612 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:11:49,614 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-07 09:11:49,615 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:11:50,372 INFO [run_ww3] running in process 23580
2026-03-07 09:11:50,373 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:11:50,374 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-07 09:11:50,376 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:11:50,380 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-07T091150.374070-0800
2026-03-07 09:11:50,381 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-07T091150.374070-0800/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-07 09:11:50,393 INFO [run_ww3] forecast WaveWatch3 run for 2026-03-07 on arbutus.cloud-nowcast started
2026-03-07 09:11:50,400 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-07 09:11:50,712 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-07 09:11:52,733 INFO [watch_ww3] running in process 23736
2026-03-07 09:11:52,736 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:11:52,737 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-07 09:11:52,738 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:11:52,768 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-07 09:12:21,936 INFO [download_wwatch3_results] nowcast 2026-03-07 results files from arbutus.cloud-nowcast downloaded
2026-03-07 09:12:21,939 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-07 09:14:18,510 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19082572 = 2026-03-07 12:34:00 UTC, 52.4% complete
2026-03-07 09:16:52,770 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete
2026-03-07 09:19:18,512 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19082765 = 2026-03-07 14:42:40 UTC, 61.3% complete
2026-03-07 09:21:52,772 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 189, 29.2% complete
2026-03-07 09:24:18,514 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19082957 = 2026-03-07 16:50:40 UTC, 70.2% complete
2026-03-07 09:26:52,773 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 282, 43.5% complete
2026-03-07 09:29:18,515 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19083148 = 2026-03-07 18:58:00 UTC, 79.1% complete
2026-03-07 09:31:52,775 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 387, 59.7% complete
2026-03-07 09:34:18,517 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19083341 = 2026-03-07 21:06:40 UTC, 88.0% complete
2026-03-07 09:36:52,776 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 480, 74.1% complete
2026-03-07 09:39:18,518 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19083532 = 2026-03-07 23:14:00 UTC, 96.9% complete
2026-03-07 09:41:52,777 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 579, 89.4% complete
2026-03-07 09:44:18,519 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19083600 = 2026-03-07 23:59:20 UTC, 100.0% complete
2026-03-07 09:46:52,778 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-07 09:49:18,528 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-03-07 09:49:18,559 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-07 09:49:19,146 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:49:21,667 INFO [download_results] running in process 1161565
2026-03-07 09:49:21,668 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:49:21,669 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-07 09:49:21,670 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:49:21,671 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/07mar26 to /results2/SalishSea/nowcast-green.202111
2026-03-07 09:51:30,731 INFO [download_results] nowcast-green 2026-03-07 results files from arbutus.cloud-nowcast downloaded
2026-03-07 09:51:30,736 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-07 09:51:31,068 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:51:31,074 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-03-07 09:51:31,079 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:51:31,085 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:51:31,092 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:51:33,352 INFO [ping_erddap] running in process 1161615
2026-03-07 09:51:33,354 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:51:33,355 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-07 09:51:33,357 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:51:33,366 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-03-07 09:51:33,370 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-07 09:51:35,749 INFO [make_averaged_dataset] running in process 1161617
2026-03-07 09:51:35,750 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:51:35,751 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-03-07 09:51:35,752 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:51:35,753 INFO [make_averaged_dataset] creating day-averaged dataset for 07-Mar-2026 biology
2026-03-07 09:51:35,833 INFO [make_averaged_dataset] running in process 1161616
2026-03-07 09:51:35,834 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:51:35,835 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-03-07 09:51:35,836 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:51:35,836 INFO [make_averaged_dataset] creating day-averaged dataset for 07-Mar-2026 physics
2026-03-07 09:51:35,843 INFO [make_averaged_dataset] running in process 1161618
2026-03-07 09:51:35,844 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:51:35,845 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-03-07 09:51:35,846 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:51:35,847 INFO [make_averaged_dataset] creating day-averaged dataset for 07-Mar-2026 chemistry
2026-03-07 09:51:37,185 INFO [make_plots] running in process 1161613
2026-03-07 09:51:37,187 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:51:37,187 INFO [make_plots] publishing log messages to tcp://*:5566
2026-03-07 09:51:37,189 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:51:37,831 INFO [make_plots] preparing render list for 2026-03-07 NEMO nowcast-green research figures
2026-03-07 09:51:41,391 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:42,213 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:43,015 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:43,819 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:44,663 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:45,461 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:46,278 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:48,893 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:49,748 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:50,635 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:51,485 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:52,273 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:52,780 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-07 09:51:52,789 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-07 09:51:53,278 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:51:53,549 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:53,550 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:53,551 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:53,552 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-03-07 NEMO nowcast-green research render list
2026-03-07 09:51:53,553 INFO [make_plots] starting to render nemo nowcast-green research 07mar26 figures
2026-03-07 09:51:54,544 INFO [download_wwatch3_results] running in process 1161757
2026-03-07 09:51:54,545 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:51:54,546 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2026-03-07 09:51:54,547 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:52:20,891 INFO [make_averaged_dataset] day-averaged dataset for 07-Mar-2026 physics created
2026-03-07 09:52:20,921 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-07 09:52:33,575 INFO [download_wwatch3_results] forecast 2026-03-07 results files from arbutus.cloud-nowcast downloaded
2026-03-07 09:52:33,585 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-07 09:52:33,927 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:52:35,261 INFO [update_forecast_datasets] running in process 1161772
2026-03-07 09:52:35,263 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:52:35,263 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2026-03-07 09:52:35,264 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:52:35,265 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/07mar26 run
2026-03-07 09:52:35,268 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/07mar26 run
2026-03-07 09:52:35,269 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/07mar26 run
2026-03-07 09:52:35,475 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/07mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-07 09:52:35,475 INFO [update_forecast_datasets] wwatch3 2026-03-07 forecast rolling forecast datasets updated
2026-03-07 09:52:35,479 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-07 09:52:35,766 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-07 09:52:37,034 INFO [ping_erddap] running in process 1161778
2026-03-07 09:52:37,038 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:52:37,038 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-07 09:52:37,039 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:52:37,041 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-07 09:52:37,043 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-07 09:52:37,385 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-03-07'], host='localhost')
2026-03-07 09:52:41,390 INFO [make_plots] running in process 1161783
2026-03-07 09:52:41,399 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 09:52:41,417 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-07 09:52:41,435 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-07 09:52:41,453 INFO [make_plots] preparing render list for 2026-03-07 WaveWatch3 forecast publish figures
2026-03-07 09:52:41,453 INFO [make_plots] added Halibut Bank figure to 2026-03-07 WaveWatch3 forecast publish render list
2026-03-07 09:52:41,454 INFO [make_plots] added Sentry Shoal figure to 2026-03-07 WaveWatch3 forecast publish render list
2026-03-07 09:52:41,455 INFO [make_plots] starting to render wwatch3 forecast publish 07mar26 figures
2026-03-07 09:52:42,087 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-07 09:52:42,965 INFO [make_averaged_dataset] day-averaged dataset for 07-Mar-2026 chemistry created
2026-03-07 09:52:42,969 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-07 09:52:46,710 INFO [make_averaged_dataset] day-averaged dataset for 07-Mar-2026 biology created
2026-03-07 09:52:46,714 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-07 09:52:49,129 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-07 09:52:53,374 INFO [make_plots] finished rendering wwatch3 forecast publish 07mar26 figures
2026-03-07 09:52:53,375 INFO [make_plots] wwatch3 publish plots for 2026-03-07 forecast completed
2026-03-07 09:52:53,379 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-07 11:02:27,656 INFO [make_plots] finished rendering nemo nowcast-green research 07mar26 figures
2026-03-07 11:02:27,682 INFO [make_plots] nemo research plots for 2026-03-07 nowcast-green completed
2026-03-07 11:02:27,762 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-07 13:44:09,163 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260307/18/
2026-03-07 13:44:09,193 INFO [collect_weather] 2026-03-07 2.5km weather forecast 18 collection complete
2026-03-07 13:44:09,272 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-07 13:44:10,057 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-03-07 13:44:10,069 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-03-07 13:44:10,075 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-03-07 13:44:10,084 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-03-08'], host='localhost')
2026-03-07 13:44:10,257 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260307/18/
2026-03-07 13:44:10,263 INFO [crop_gribs] 2026-03-07 18 GRIBs cropping complete
2026-03-07 13:44:10,273 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-07 13:44:12,768 INFO [download_weather] running in process 1164235
2026-03-07 13:44:12,769 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 13:44:12,770 INFO [download_weather] publishing log messages to tcp://*:5562
2026-03-07 13:44:12,771 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-07 13:44:12,771 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260307
2026-03-07 13:44:12,772 INFO [collect_weather] running in process 1164237
2026-03-07 13:44:12,775 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 13:44:12,776 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-03-07 13:44:12,777 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-07 13:44:12,778 INFO [download_weather] running in process 1164236
2026-03-07 13:44:12,779 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 13:44:12,779 INFO [download_weather] publishing log messages to tcp://*:5564
2026-03-07 13:44:12,780 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-07 13:44:12,781 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260307
2026-03-07 13:44:12,837 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-03-07 13:44:14,387 INFO [crop_gribs] running in process 1164238
2026-03-07 13:44:14,388 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 13:44:14,390 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-03-07 13:44:14,391 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-07 13:44:14,392 INFO [crop_gribs] cropping 2026-03-08 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-03-07 13:44:14,404 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260308/00/
2026-03-07 13:47:42,396 INFO [download_weather] 2026-03-07 1km weather forecast 00 downloads complete
2026-03-07 13:47:42,399 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-07 13:47:46,378 INFO [download_weather] 2026-03-07 1km weather forecast 12 downloads complete
2026-03-07 13:47:46,380 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-07 19:44:34,343 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260308/00/
2026-03-07 19:44:34,380 INFO [collect_weather] 2026-03-08 2.5km weather forecast 00 collection complete
2026-03-07 19:44:34,476 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-07 19:44:35,335 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-03-07 19:44:35,352 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-03-08'], host='localhost')
2026-03-07 19:44:38,194 INFO [collect_weather] running in process 1168807
2026-03-07 19:44:38,194 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 19:44:38,195 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-07 19:44:38,196 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-07 19:44:38,239 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-03-07 19:44:39,923 INFO [crop_gribs] running in process 1168808
2026-03-07 19:44:39,924 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-07 19:44:39,926 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-03-07 19:44:39,927 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-07 19:44:39,928 INFO [crop_gribs] cropping 2026-03-08 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-03-07 19:44:39,950 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260308/06/
2026-03-07 21:44:15,994 INFO [crop_gribs] crop_gribs 00 has watched for 8h; retrying remaining 1 file(s)
2026-03-07 21:44:17,414 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260308/00/
2026-03-07 21:44:17,420 INFO [crop_gribs] 2026-03-08 00 GRIBs cropping complete
2026-03-07 21:44:17,499 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-08 01:42:25,052 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260308/06/
2026-03-08 01:42:25,076 INFO [collect_weather] 2026-03-08 2.5km weather forecast 06 collection complete
2026-03-08 01:42:25,098 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-08 01:42:25,693 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,706 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,711 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,723 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,728 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,732 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,736 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,740 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,748 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,755 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,760 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,764 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,772 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,779 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,790 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-03-08 01:42:25,796 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-03-08 01:42:25,815 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-03-08 01:42:25,821 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-03-07'], host='localhost')
2026-03-08 01:42:25,836 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-03-08 01:42:25,843 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-03-08 01:42:25,861 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-03-08 01:42:26,051 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260308/06/
2026-03-08 01:42:26,076 INFO [crop_gribs] 2026-03-08 06 GRIBs cropping complete
2026-03-08 01:42:26,112 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-08 01:42:26,955 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-03-08 01:42:29,121 INFO [collect_weather] running in process 1173196
2026-03-08 01:42:29,122 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:29,123 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-08 01:42:29,124 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:29,130 INFO [collect_NeahBay_ssh] running in process 1173195
2026-03-08 01:42:29,131 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:29,134 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-03-08 01:42:29,135 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:29,136 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-03-08
2026-03-08 01:42:29,186 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-03-08 01:42:30,413 INFO [collect_river_data] running in process 1173179
2026-03-08 01:42:30,414 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,414 INFO [collect_river_data] publishing log messages to tcp://*:5564
2026-03-08 01:42:30,415 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,416 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-03-07
2026-03-08 01:42:30,486 INFO [collect_river_data] running in process 1173190
2026-03-08 01:42:30,487 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,488 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-08 01:42:30,489 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,490 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-03-07
2026-03-08 01:42:30,516 INFO [collect_river_data] running in process 1173185
2026-03-08 01:42:30,518 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,518 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-08 01:42:30,519 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,520 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-03-07
2026-03-08 01:42:30,524 INFO [collect_river_data] running in process 1173176
2026-03-08 01:42:30,525 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,526 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-08 01:42:30,526 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,528 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-03-07
2026-03-08 01:42:30,532 INFO [collect_river_data] running in process 1173182
2026-03-08 01:42:30,532 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,533 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-08 01:42:30,534 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,535 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-03-07
2026-03-08 01:42:30,540 INFO [collect_river_data] running in process 1173178
2026-03-08 01:42:30,541 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,542 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-03-08 01:42:30,543 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,544 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-03-07
2026-03-08 01:42:30,551 INFO [collect_river_data] running in process 1173177
2026-03-08 01:42:30,552 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,553 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-03-08 01:42:30,553 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,554 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-03-07
2026-03-08 01:42:30,561 INFO [collect_river_data] running in process 1173186
2026-03-08 01:42:30,562 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,564 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-03-08 01:42:30,568 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,569 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-03-07
2026-03-08 01:42:30,570 INFO [collect_river_data] running in process 1173187
2026-03-08 01:42:30,571 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,573 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-03-08 01:42:30,575 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,578 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-03-07
2026-03-08 01:42:30,583 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-03-07 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-03-08 01:42:30,585 INFO [collect_river_data] ECCC Capilano river data collection for 2026-03-07 completed
2026-03-08 01:42:30,588 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:30,589 INFO [collect_river_data] running in process 1173189
2026-03-08 01:42:30,590 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,592 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-03-08 01:42:30,593 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,595 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-03-07
2026-03-08 01:42:30,612 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-03-08 01:42:30,613 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-03-07 completed
2026-03-08 01:42:30,619 INFO [collect_river_data] running in process 1173181
2026-03-08 01:42:30,620 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,620 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-03-08 01:42:30,622 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,623 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-03-07
2026-03-08 01:42:30,624 INFO [collect_river_data] running in process 1173183
2026-03-08 01:42:30,625 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-03-08 01:42:30,627 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,628 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-03-07 completed
2026-03-08 01:42:30,628 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-03-08 01:42:30,629 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,630 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-03-07
2026-03-08 01:42:30,631 INFO [collect_river_data] running in process 1173188
2026-03-08 01:42:30,634 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,635 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-03-08 01:42:30,635 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-03-08 01:42:30,636 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-03-08 01:42:30,637 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-03-08 01:42:30,638 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-03-07 completed
2026-03-08 01:42:30,639 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-03-07 completed
2026-03-08 01:42:30,640 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-03-07 completed
2026-03-08 01:42:30,641 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,643 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-03-07
2026-03-08 01:42:30,644 INFO [collect_river_data] running in process 1173180
2026-03-08 01:42:30,646 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,647 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-03-08 01:42:30,648 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-03-08 01:42:30,649 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-03-07 completed
2026-03-08 01:42:30,649 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,651 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-03-07
2026-03-08 01:42:30,654 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-03-08 01:42:30,654 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-03-08 01:42:30,655 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-03-07 completed
2026-03-08 01:42:30,656 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-03-07 completed
2026-03-08 01:42:30,664 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-03-08 01:42:30,664 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-03-07 completed
2026-03-08 01:42:30,667 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-03-08 01:42:30,671 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-03-07 completed
2026-03-08 01:42:30,673 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-03-07 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-03-08 01:42:30,673 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-03-07 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-03-08 01:42:30,674 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-03-07 completed
2026-03-08 01:42:30,675 INFO [collect_river_data] ECCC Fraser river data collection for 2026-03-07 completed
2026-03-08 01:42:30,722 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-03-07 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-03-08 01:42:30,723 INFO [collect_river_data] ECCC Englishman river data collection for 2026-03-07 completed
2026-03-08 01:42:30,868 INFO [grib_to_netcdf] running in process 1173259
2026-03-08 01:42:30,869 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:30,870 INFO [grib_to_netcdf] publishing log messages to tcp://*:5578
2026-03-08 01:42:30,871 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:30,872 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-08 forecast2 run
2026-03-08 01:42:30,972 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:31,063 INFO [get_vfpa_hadcp] running in process 1173194
2026-03-08 01:42:31,065 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:31,066 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2026-03-08 01:42:31,067 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:31,068 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-07
2026-03-08 01:42:31,174 INFO [crop_gribs] running in process 1173197
2026-03-08 01:42:31,177 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:31,180 INFO [crop_gribs] publishing log messages to tcp://*:5590
2026-03-08 01:42:31,182 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:31,184 INFO [crop_gribs] cropping 2026-03-08 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-03-08 01:42:31,194 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260308/12/
2026-03-08 01:42:31,320 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:31,771 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:31,866 INFO [collect_NeahBay_ssh] 2026-03-08T00:00:00-08:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-03-08 01:42:32,043 INFO [get_onc_ctd] running in process 1173191
2026-03-08 01:42:32,044 INFO [get_onc_ferry] running in process 1173193
2026-03-08 01:42:32,045 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:32,045 INFO [get_onc_ctd] publishing log messages to tcp://*:5566
2026-03-08 01:42:32,046 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:32,047 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-03-07
2026-03-08 01:42:32,047 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:32,048 INFO [get_onc_ferry] publishing log messages to tcp://*:5567
2026-03-08 01:42:32,049 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:32,049 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-03-07
2026-03-08 01:42:32,055 INFO [get_onc_ctd] running in process 1173192
2026-03-08 01:42:32,057 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:32,057 INFO [get_onc_ctd] publishing log messages to tcp://*:5568
2026-03-08 01:42:32,058 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:32,059 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-03-07
2026-03-08 01:42:32,186 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:32,540 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:32,864 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:33,203 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:33,548 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:33,876 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:33,879 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-07 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202603.nc
2026-03-08 01:42:33,879 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-03 netcdf file
2026-03-08 01:42:34,177 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:34,501 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:34,802 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:35,138 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-08 01:42:35,781 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-08 01:42:36,160 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 01:42:36,169 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2026-03-08 01:42:36,675 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2026-03-08 01:42:38,009 INFO [ping_erddap] running in process 1173700
2026-03-08 01:42:38,010 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:38,011 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-08 01:42:38,012 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:38,013 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2026-03-08 01:42:38,015 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 01:42:40,134 INFO [make_ssh_files] running in process 1173687
2026-03-08 01:42:40,135 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:40,136 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-03-08 01:42:40,137 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:40,138 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-03-08 NOAA Neah Bay 00Z observation and forecast values
2026-03-08 01:42:42,371 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d03.nc
2026-03-08 01:42:42,612 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d04.nc
2026-03-08 01:42:42,815 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d05.nc
2026-03-08 01:42:43,040 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d06.nc
2026-03-08 01:42:43,251 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d09.nc
2026-03-08 01:42:43,272 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d07.nc
2026-03-08 01:42:43,514 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d08.nc
2026-03-08 01:42:43,743 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d09.nc
2026-03-08 01:42:43,955 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d10.nc
2026-03-08 01:42:44,157 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d11.nc
2026-03-08 01:42:44,326 INFO [make_ssh_files] sea surface height boundary file for 2026-03-08 forecast2 run created
2026-03-08 01:42:44,329 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-08 01:42:44,682 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-08 01:42:44,686 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-08 01:42:46,135 INFO [make_201702_runoff_file] running in process 1173756
2026-03-08 01:42:46,137 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:46,138 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2026-03-08 01:42:46,141 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:46,177 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-08 01:42:46,288 INFO [make_201702_runoff_file] 2026-03-08 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-08 01:42:46,290 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-08 01:42:46,739 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d10.nc
2026-03-08 01:42:46,745 INFO [grib_to_netcdf] 2026-03-08 NEMO-atmos forcing file for forecast2 created
2026-03-08 01:42:46,935 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-08 01:42:46,962 INFO [make_runoff_file] running in process 1173754
2026-03-08 01:42:46,963 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:46,964 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2026-03-08 01:42:46,965 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:46,969 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-07
2026-03-08 01:42:48,095 ERROR [make_runoff_file] no 2026-03-07 discharge obs for SkagitMountVernon: patching
2026-03-08 01:42:48,868 ERROR [make_runoff_file] no 2026-03-07 discharge obs for SnohomishMonroe: patching
2026-03-08 01:42:49,138 ERROR [make_runoff_file] no 2026-03-07 discharge obs for NisquallyMcKenna: patching
2026-03-08 01:42:49,264 ERROR [make_runoff_file] no 2026-03-07 discharge obs for GreenwaterGreenwater: patching
2026-03-08 01:42:50,813 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-07: /results/forcing/rivers/R202108Dailies_y2026m03d07.nc
2026-03-08 01:42:50,814 INFO [make_runoff_file] 2026-03-07 runoff file creation completed
2026-03-08 01:42:50,817 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-08 01:42:51,113 INFO [get_onc_ctd] 2026-03-07 ONC SEVIP CTD T&S file created
2026-03-08 01:42:51,114 INFO [get_onc_ctd] 2026-03-07 ONC SCVIP CTD T&S file created
2026-03-08 01:42:51,133 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-03-08 01:42:51,139 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-03-08 01:42:51,143 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-03-08 01:42:51,146 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-03-08 01:42:51,153 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-08 01:42:51,595 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-03-08 01:42:51,600 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-08 01:42:51,775 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-03-07
2026-03-08 01:42:52,007 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-03-08 01:42:52,743 INFO [upload_forcing] running in process 1173804
2026-03-08 01:42:52,744 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:52,746 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-03-08 01:42:52,746 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:52,754 INFO [upload_forcing] running in process 1173803
2026-03-08 01:42:52,755 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:52,756 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-08 01:42:52,758 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:52,786 INFO [upload_forcing] running in process 1173801
2026-03-08 01:42:52,787 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:52,788 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-08 01:42:52,789 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:52,796 INFO [upload_forcing] running in process 1173802
2026-03-08 01:42:52,797 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:52,798 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-08 01:42:52,799 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:52,878 INFO [ping_erddap] running in process 1173805
2026-03-08 01:42:52,879 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:52,880 INFO [ping_erddap] publishing log messages to tcp://*:5568
2026-03-08 01:42:52,881 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:52,883 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-03-08 01:42:52,885 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 01:42:53,174 INFO [ping_erddap] running in process 1173807
2026-03-08 01:42:53,176 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:53,177 INFO [ping_erddap] publishing log messages to tcp://*:5569
2026-03-08 01:42:53,178 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:53,179 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-03-08 01:42:53,229 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 01:42:54,142 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d07.nc
2026-03-08 01:42:54,189 INFO [upload_forcing] forecast2 2026-03-08 forcing files upload to optimum-hindcast completed
2026-03-08 01:42:54,191 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 01:42:55,654 INFO [upload_forcing] forecast2 2026-03-08 forcing files upload to arbutus.cloud-nowcast completed
2026-03-08 01:42:55,659 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 01:42:55,989 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-08'], host='localhost')
2026-03-08 01:42:57,300 INFO [make_forcing_links] running in process 1173843
2026-03-08 01:42:57,301 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:42:57,302 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-03-08 01:42:57,303 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:42:58,449 INFO [make_forcing_links] forecast2 2026-03-08 forcing file links on arbutus.cloud-nowcast created
2026-03-08 01:42:58,453 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-08 01:42:58,782 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 01:42:59,544 INFO [upload_forcing] forecast2 2026-03-08 forcing files upload to orcinus-nowcast-agrif completed
2026-03-08 01:42:59,550 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 01:43:01,026 INFO [run_NEMO] running in process 4066
2026-03-08 01:43:01,027 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:43:01,028 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-08 01:43:01,089 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:43:01,765 INFO [upload_forcing] forecast2 2026-03-08 forcing files upload to robot.nibi completed
2026-03-08 01:43:01,770 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-08 01:43:01,843 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-08 01:43:02,150 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/07mar26forecast2_2026-03-08T014301.153468-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-08 01:43:02,164 INFO [run_NEMO] forecast2 NEMO run for 2026-03-08 on arbutus.cloud-nowcast started
2026-03-08 01:43:02,168 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-08 01:43:02,460 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-08 01:43:04,686 INFO [watch_NEMO] running in process 4342
2026-03-08 01:43:04,693 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:43:04,695 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-08 01:43:04,697 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:43:04,701 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-08 01:43:14,185 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-03-07
2026-03-08 01:43:35,366 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-03-07
2026-03-08 01:43:56,987 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-03-07
2026-03-08 01:43:57,057 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,064 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,066 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,067 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-03-07
2026-03-08 01:43:57,146 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,152 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,154 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,156 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-03-07
2026-03-08 01:43:57,222 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,225 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,227 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-03-07
2026-03-08 01:43:57,285 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,288 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,290 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-03-07
2026-03-08 01:43:57,358 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,361 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-07; substituting empty dataset
2026-03-08 01:43:57,557 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-03-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-08 01:43:57,573 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-03-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-08 01:43:57,587 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-03-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-08 01:43:57,603 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-03-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-08 01:43:57,617 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-03-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-08 01:43:57,634 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-03-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-08 01:43:57,649 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-08 01:44:31,592 INFO [get_onc_ferry] 2026-03-07 ONC TWDP ferry data file created
2026-03-08 01:44:31,594 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-03-08 01:44:31,908 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-03-08 01:44:33,173 INFO [ping_erddap] running in process 1173868
2026-03-08 01:44:33,175 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 01:44:33,176 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-08 01:44:33,177 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 01:44:33,179 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-03-08 01:44:33,182 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 01:48:04,589 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19086244 = 2026-03-09 05:22:00 UTC, 17.9% complete
2026-03-08 01:53:04,590 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19086819 = 2026-03-09 11:45:20 UTC, 39.2% complete
2026-03-08 01:58:04,592 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19087379 = 2026-03-09 17:58:40 UTC, 59.9% complete
2026-03-08 03:03:04,593 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19087913 = 2026-03-09 23:54:40 UTC, 79.7% complete
2026-03-08 03:08:04,594 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19088414 = 2026-03-10 05:28:40 UTC, 98.3% complete
2026-03-08 03:13:04,598 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-03-08 03:13:04,602 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-08 03:13:05,160 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 03:13:05,173 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 03:13:05,340 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-07'], host='localhost')
2026-03-08 03:13:06,509 INFO [download_results] running in process 1174225
2026-03-08 03:13:06,510 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:13:06,512 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-08 03:13:06,513 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:13:06,514 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/07mar26 to /results/SalishSea/forecast2.202111
2026-03-08 03:13:07,967 INFO [make_ww3_current_file] running in process 4865
2026-03-08 03:13:07,969 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:13:07,971 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-08 03:13:07,972 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:13:07,974 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-08 forecast2 run
2026-03-08 03:13:07,995 INFO [make_ww3_wind_file] running in process 4866
2026-03-08 03:13:07,997 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:13:07,998 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-08 03:13:08,000 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:13:08,001 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-08 forecast2 run
2026-03-08 03:13:16,760 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-08 forecast2 run
2026-03-08 03:13:16,765 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-08 03:13:49,637 INFO [download_results] forecast2 2026-03-07 results files from arbutus.cloud-nowcast downloaded
2026-03-08 03:13:49,641 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-08 03:13:49,988 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-03-07'], host='localhost')
2026-03-08 03:13:52,142 INFO [make_CHS_currents_file] running in process 1174241
2026-03-08 03:13:52,144 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:13:52,145 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-08 03:13:52,145 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:14:27,635 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-07 for forecast2
2026-03-08 03:14:27,638 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-08 03:14:27,953 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-03-07'], host='localhost')
2026-03-08 03:14:29,325 INFO [update_forecast_datasets] running in process 1174272
2026-03-08 03:14:29,326 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:14:29,328 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-08 03:14:29,329 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:14:29,330 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/07mar26 run
2026-03-08 03:16:46,981 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-08 forecast2 run
2026-03-08 03:16:46,986 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-08 03:16:47,310 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-08'], host='arbutus.cloud-nowcast')
2026-03-08 03:16:49,339 INFO [run_ww3] running in process 5447
2026-03-08 03:16:49,340 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:16:49,342 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-08 03:16:49,343 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:16:49,348 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-08T031649.335826-0700
2026-03-08 03:16:49,349 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-08T031649.335826-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-08 03:16:49,357 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-03-08 on arbutus.cloud-nowcast started
2026-03-08 03:16:49,361 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-08 03:16:49,679 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-08 03:16:51,675 INFO [watch_ww3] running in process 5534
2026-03-08 03:16:51,676 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:16:51,677 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-08 03:16:51,679 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:16:51,711 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-08 03:18:44,831 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/07mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-08 03:18:44,832 INFO [update_forecast_datasets] nemo 2026-03-07 forecast2 rolling forecast datasets updated
2026-03-08 03:18:44,835 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-08 03:18:45,115 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-08 03:18:45,121 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-03-07'], host='localhost')
2026-03-08 03:18:45,125 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-03-07'], host='localhost')
2026-03-08 03:18:46,390 INFO [ping_erddap] running in process 1174373
2026-03-08 03:18:46,391 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:18:46,392 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-08 03:18:46,392 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:18:46,400 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-08 03:18:46,404 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-08 03:18:47,819 INFO [make_surface_current_tiles] running in process 1174376
2026-03-08 03:18:47,820 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:18:47,821 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-08 03:18:47,822 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:18:47,824 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-07 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/07mar26
2026-03-08 03:18:48,812 INFO [make_plots] running in process 1174375
2026-03-08 03:18:48,813 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:18:48,814 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-08 03:18:48,824 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:18:49,406 INFO [make_plots] preparing render list for 2026-03-07 NEMO forecast2 publish figures
2026-03-08 03:18:49,832 INFO [make_plots] added Website_thumbnail figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,833 INFO [make_plots] added Threshold_website figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,834 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,834 INFO [make_plots] added Boundary Bay figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,835 INFO [make_plots] added Campbell River figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,836 INFO [make_plots] added Cherry Point figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,837 INFO [make_plots] added Friday Harbor figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,837 INFO [make_plots] added Halfmoon Bay figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,838 INFO [make_plots] added Nanaimo figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,839 INFO [make_plots] added Neah Bay figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,840 INFO [make_plots] added New Westminster figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,840 INFO [make_plots] added Patricia Bay figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,841 INFO [make_plots] added Point Atkinson figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,842 INFO [make_plots] added Port Renfrew figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,843 INFO [make_plots] added Sand Heads figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,843 INFO [make_plots] added Sandy Cove figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,844 INFO [make_plots] added Squamish figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,845 INFO [make_plots] added Victoria figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,846 INFO [make_plots] added Woodwards Landing figure to 2026-03-07 NEMO forecast2 publish render list
2026-03-08 03:18:49,846 INFO [make_plots] starting to render nemo forecast2 publish 07mar26 figures
2026-03-08 03:21:51,711 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 84, 15.6% complete
2026-03-08 03:26:51,712 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 183, 33.9% complete
2026-03-08 03:27:02,950 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-08 03:27:02,950 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-08 03:27:25,284 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:28:27,930 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-08 03:28:27,931 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-08 03:28:50,512 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:29:33,214 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:29:45,273 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/07mar26
2026-03-08 03:29:50,365 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:30:07,972 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:30:25,252 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:30:42,310 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:30:59,668 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:31:16,312 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-08 03:31:16,312 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-08 03:31:33,228 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:31:49,943 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-03 00:00:00Z to 2026-03-10 05:55:00Z
2026-03-08 03:31:51,714 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 282, 52.2% complete
2026-03-08 03:32:06,115 INFO [make_plots] finished rendering nemo forecast2 publish 07mar26 figures
2026-03-08 03:32:06,118 INFO [make_plots] nemo publish plots for 2026-03-07 forecast2 completed
2026-03-08 03:32:06,126 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-08 03:32:06,412 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-03-07'], host='localhost')
2026-03-08 03:32:09,613 INFO [make_feeds] running in process 1179406
2026-03-08 03:32:09,614 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:32:09,615 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-08 03:32:09,617 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:32:24,036 INFO [make_feeds] ATOM feeds for 2026-03-07 forecast2 run completed
2026-03-08 03:32:24,039 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-08 03:32:24,365 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-03-08 03:32:25,473 INFO [clear_checklist] running in process 1179428
2026-03-08 03:32:25,474 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:32:25,475 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-03-08 03:32:25,476 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:32:25,477 INFO [clear_checklist] requesting that manager clear system state checklist
2026-03-08 03:32:25,480 INFO [manager] writing checklist to log file
2026-03-08 03:32:25,524 INFO [manager] checklist cleared
2026-03-08 03:32:25,526 INFO [clear_checklist] nowcast system checklist cleared
2026-03-08 03:32:25,529 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-03-08 03:32:26,651 INFO [rotate_logs] running in process 1179433
2026-03-08 03:32:26,652 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-08 03:32:26,653 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-03-08 03:32:26,654 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-03-08 03:32:26,654 INFO [rotate_logs] rotating log files