2026-03-09 05:34:40,788 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-03-09 05:34:40,796 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-03-09 05:34:40,802 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-03-09 05:34:40,802 INFO [rotate_logs] log files rotated
2026-03-09 05:34:40,806 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-03-09 05:36:37,389 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/08mar26
2026-03-09 05:36:37,412 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-08 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/08mar26
2026-03-09 05:36:37,414 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-08 forecast2 completed
2026-03-09 05:36:37,701 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-09 05:39:20,295 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 381, 70.6% complete
2026-03-09 05:44:20,295 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 480, 88.9% complete
2026-03-09 05:49:20,297 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-09 05:49:20,300 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-09 05:49:20,676 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 05:49:21,979 INFO [download_wwatch3_results] running in process 1209496
2026-03-09 05:49:21,980 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:49:21,981 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-09 05:49:21,983 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:49:58,811 INFO [download_wwatch3_results] forecast2 2026-03-09 results files from arbutus.cloud-nowcast downloaded
2026-03-09 05:49:58,814 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-09 05:49:58,962 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 05:50:00,285 INFO [update_forecast_datasets] running in process 1209511
2026-03-09 05:50:00,286 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:50:00,288 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-09 05:50:00,289 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:50:00,290 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/09mar26 run
2026-03-09 05:50:00,294 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/09mar26 run
2026-03-09 05:50:00,294 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/09mar26 run
2026-03-09 05:50:22,423 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/09mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-09 05:50:22,424 INFO [update_forecast_datasets] wwatch3 2026-03-09 forecast2 rolling forecast datasets updated
2026-03-09 05:50:22,427 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-09 05:50:22,578 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-09 05:50:23,821 INFO [ping_erddap] running in process 1209524
2026-03-09 05:50:23,822 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:50:23,823 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-09 05:50:23,824 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:50:23,826 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-09 05:50:23,828 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 05:50:23,977 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 05:50:27,760 INFO [make_plots] running in process 1209529
2026-03-09 05:50:27,761 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 05:50:27,762 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-09 05:50:27,763 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-09 05:50:27,764 INFO [make_plots] preparing render list for 2026-03-09 WaveWatch3 forecast2 publish figures
2026-03-09 05:50:27,765 INFO [make_plots] added Halibut Bank figure to 2026-03-09 WaveWatch3 forecast2 publish render list
2026-03-09 05:50:27,766 INFO [make_plots] added Sentry Shoal figure to 2026-03-09 WaveWatch3 forecast2 publish render list
2026-03-09 05:50:27,767 INFO [make_plots] starting to render wwatch3 forecast2 publish 09mar26 figures
2026-03-09 05:50:28,301 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-09 05:50:33,626 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-09 05:50:38,096 INFO [make_plots] finished rendering wwatch3 forecast2 publish 09mar26 figures
2026-03-09 05:50:38,097 INFO [make_plots] wwatch3 publish plots for 2026-03-09 forecast2 completed
2026-03-09 05:50:38,101 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-09 09:08:00,212 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260309/12/
2026-03-09 09:08:00,213 INFO [collect_weather] 2026-03-09 2.5km weather forecast 12 collection complete
2026-03-09 09:08:00,217 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-09 09:08:00,540 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 09:08:00,548 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 09:08:00,554 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 09:08:00,558 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-03-08'], host='localhost')
2026-03-09 09:08:00,562 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-03-09 09:08:00,567 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-03-09 09:08:00,573 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-03-09 09:08:00,581 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-03-09 09:08:00,588 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-03-09 09:08:02,285 INFO [collect_NeahBay_ssh] running in process 1212106
2026-03-09 09:08:02,286 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:02,287 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2026-03-09 09:08:02,288 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:02,290 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-03-09
2026-03-09 09:08:02,292 INFO [collect_weather] running in process 1212108
2026-03-09 09:08:02,292 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:02,293 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-03-09 09:08:02,293 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:02,309 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-03-09 09:08:02,428 INFO [download_live_ocean] running in process 1212107
2026-03-09 09:08:02,429 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:02,430 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2026-03-09 09:08:02,431 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:02,432 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-03-09
2026-03-09 09:08:02,592 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260309/12/
2026-03-09 09:08:02,594 INFO [crop_gribs] 2026-03-09 12 GRIBs cropping complete
2026-03-09 09:08:02,597 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-09 09:08:02,882 INFO [collect_river_data] running in process 1212100
2026-03-09 09:08:02,887 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:02,893 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-09 09:08:02,899 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:02,902 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-03-08
2026-03-09 09:08:02,910 INFO [collect_river_data] running in process 1212103
2026-03-09 09:08:02,912 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:02,914 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-09 09:08:02,915 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:02,917 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-03-08
2026-03-09 09:08:02,920 INFO [collect_river_data] running in process 1212102
2026-03-09 09:08:02,924 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:02,925 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-09 09:08:02,926 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:02,927 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-03-08
2026-03-09 09:08:02,929 INFO [collect_river_data] running in process 1212104
2026-03-09 09:08:02,931 INFO [download_live_ocean] downloaded 10211572 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.03.09/ubc.nc to /results/forcing/LiveOcean/downloaded/20260309/low_passed_UBC.nc
2026-03-09 09:08:02,932 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:02,933 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-03-09 09:08:02,935 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-03-09 09:08:02,936 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-09 09:08:02,937 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:02,939 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-03-08
2026-03-09 09:08:03,031 INFO [make_turbidity_file] running in process 1212105
2026-03-09 09:08:03,032 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:03,033 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-03-09 09:08:03,034 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:03,035 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-03-09
2026-03-09 09:08:03,450 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-03-08&endDT=2026-03-08 "HTTP/1.1 200 "
2026-03-09 09:08:03,453 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-03-08&endDT=2026-03-08 "HTTP/1.1 200 "
2026-03-09 09:08:03,462 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-03-09 09:08:03,462 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-03-09 09:08:03,463 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-03-08 completed
2026-03-09 09:08:03,464 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-03-08 completed
2026-03-09 09:08:03,466 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-03-08&endDT=2026-03-08 "HTTP/1.1 200 "
2026-03-09 09:08:03,469 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-03-08&endDT=2026-03-08 "HTTP/1.1 200 "
2026-03-09 09:08:03,470 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 09:08:03,477 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-03-09 09:08:03,478 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-03-08 completed
2026-03-09 09:08:03,479 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-03-08 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-03-09 09:08:03,481 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-03-08 completed
2026-03-09 09:08:03,483 INFO [crop_gribs] running in process 1212109
2026-03-09 09:08:03,484 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:03,486 INFO [crop_gribs] publishing log messages to tcp://*:5570
2026-03-09 09:08:03,487 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:03,489 INFO [crop_gribs] cropping 2026-03-09 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-03-09 09:08:03,495 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260309/18/
2026-03-09 09:08:03,628 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 09:08:03,787 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 09:08:03,961 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-09 09:08:03,971 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260309/low_passed_UBC.nc
2026-03-09 09:08:03,972 INFO [download_live_ocean] 2026-03-09 Live Ocean file for Salish Sea western boundary downloaded
2026-03-09 09:08:04,115 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-03-09 09:08:04,407 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-03-09'], host='localhost')
2026-03-09 09:08:04,847 INFO [collect_NeahBay_ssh] 2026-03-09T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-03-09 09:08:04,849 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-09 09:08:04,948 INFO [grib_to_netcdf] running in process 1212216
2026-03-09 09:08:04,949 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:04,950 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2026-03-09 09:08:04,951 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:04,951 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-09 nowcast and forecast runs
2026-03-09 09:08:05,048 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 09:08:05,552 INFO [make_turbidity_file] 2026-03-09 Fraser River turbidity file creation complete
2026-03-09 09:08:05,554 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-03-09 09:08:06,891 INFO [make_live_ocean_files] running in process 1212244
2026-03-09 09:08:06,892 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:06,893 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2026-03-09 09:08:06,895 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:06,896 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-03-09 Live Ocean run
2026-03-09 09:08:07,884 INFO [make_ssh_files] running in process 1212260
2026-03-09 09:08:07,885 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:07,887 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2026-03-09 09:08:07,888 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:07,890 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-03-09 NOAA Neah Bay 06Z observation and forecast values
2026-03-09 09:08:09,660 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d05.nc
2026-03-09 09:08:09,844 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d06.nc
2026-03-09 09:08:10,027 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d07.nc
2026-03-09 09:08:10,208 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d08.nc
2026-03-09 09:08:10,413 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d09.nc
2026-03-09 09:08:10,606 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d10.nc
2026-03-09 09:08:10,786 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d11.nc
2026-03-09 09:08:10,967 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d12.nc
2026-03-09 09:08:11,081 INFO [make_ssh_files] sea surface height boundary file for 2026-03-09 nowcast run created
2026-03-09 09:08:11,084 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-09 09:08:11,253 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-09 09:08:11,259 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-09 09:08:12,538 INFO [make_201702_runoff_file] running in process 1212329
2026-03-09 09:08:12,540 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:12,540 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5565
2026-03-09 09:08:12,541 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:12,568 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-09 09:08:12,661 INFO [make_201702_runoff_file] 2026-03-09 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-09 09:08:12,664 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-09 09:08:13,264 INFO [make_runoff_file] running in process 1212327
2026-03-09 09:08:13,265 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:13,266 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2026-03-09 09:08:13,267 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:13,269 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-08
2026-03-09 09:08:15,852 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-08: /results/forcing/rivers/R202108Dailies_y2026m03d08.nc
2026-03-09 09:08:15,855 INFO [make_runoff_file] 2026-03-08 runoff file creation completed
2026-03-09 09:08:15,858 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-09 09:08:16,693 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m03d09.nc
2026-03-09 09:08:25,757 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d10.nc
2026-03-09 09:08:31,133 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d11.nc
2026-03-09 09:08:31,147 INFO [grib_to_netcdf] 2026-03-09 NEMO-atmos forcing file for nowcast+ created
2026-03-09 09:08:31,150 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-09 09:08:44,184 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d09.nc
2026-03-09 09:08:44,185 INFO [make_live_ocean_files] 2026-03-09 Live Ocean western boundary conditions files created
2026-03-09 09:08:44,187 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-03-09 09:08:44,349 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-03-09 09:08:44,354 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-03-09 09:08:44,362 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-03-09 09:08:44,367 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-03-09 09:08:44,372 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-03-09 09:08:45,491 INFO [ping_erddap] running in process 1212405
2026-03-09 09:08:45,492 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:45,493 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-09 09:08:45,494 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:45,495 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-03-09 09:08:45,497 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 09:08:45,729 INFO [upload_forcing] running in process 1212410
2026-03-09 09:08:45,730 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:45,731 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-09 09:08:45,732 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:45,762 INFO [upload_forcing] running in process 1212408
2026-03-09 09:08:45,763 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:45,764 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-09 09:08:45,765 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:45,767 INFO [upload_forcing] running in process 1212407
2026-03-09 09:08:45,768 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:45,769 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-09 09:08:45,770 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:45,784 INFO [upload_forcing] running in process 1212409
2026-03-09 09:08:45,785 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:45,786 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-09 09:08:45,787 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:47,744 INFO [upload_forcing] nowcast+ 2026-03-09 forcing files upload to optimum-hindcast completed
2026-03-09 09:08:47,746 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 09:08:49,200 INFO [upload_forcing] nowcast+ 2026-03-09 forcing files upload to arbutus.cloud-nowcast completed
2026-03-09 09:08:49,202 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 09:08:49,438 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 09:08:50,663 INFO [make_forcing_links] running in process 1212441
2026-03-09 09:08:50,664 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:50,665 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-03-09 09:08:50,666 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:51,811 INFO [make_forcing_links] nowcast+ 2026-03-09 forcing file links on arbutus.cloud-nowcast created
2026-03-09 09:08:51,814 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-09 09:08:51,967 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 09:08:53,249 INFO [upload_forcing] nowcast+ 2026-03-09 forcing files upload to orcinus-nowcast-agrif completed
2026-03-09 09:08:53,253 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 09:08:54,050 INFO [run_NEMO] running in process 28731
2026-03-09 09:08:54,051 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:54,053 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-09 09:08:54,062 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:54,785 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-09 09:08:55,093 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/09mar26nowcast_2026-03-09T090854.099935-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-09 09:08:55,108 INFO [run_NEMO] nowcast NEMO run for 2026-03-09 on arbutus.cloud-nowcast started
2026-03-09 09:08:55,112 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-09 09:08:55,271 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-09 09:08:56,033 INFO [upload_forcing] nowcast+ 2026-03-09 forcing files upload to robot.nibi completed
2026-03-09 09:08:56,038 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 09:08:57,323 INFO [watch_NEMO] running in process 29005
2026-03-09 09:08:57,324 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:08:57,325 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-09 09:08:57,327 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:08:57,401 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-09 09:13:57,398 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19086250 = 2026-03-09 05:26:00 UTC, 22.6% complete
2026-03-09 09:18:57,399 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19086765 = 2026-03-09 11:09:20 UTC, 46.5% complete
2026-03-09 09:23:57,400 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19087305 = 2026-03-09 17:09:20 UTC, 71.5% complete
2026-03-09 09:28:57,402 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19087828 = 2026-03-09 22:58:00 UTC, 95.7% complete
2026-03-09 09:33:57,406 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-03-09 09:33:57,409 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-09 09:33:57,731 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 09:33:57,737 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 09:33:59,005 INFO [download_results] running in process 1212775
2026-03-09 09:33:59,006 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:33:59,008 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-09 09:33:59,009 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:33:59,010 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/09mar26 to /results/SalishSea/nowcast-blue.202111
2026-03-09 09:33:59,108 INFO [make_forcing_links] running in process 1212774
2026-03-09 09:33:59,110 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:33:59,111 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-09 09:33:59,112 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:34:00,113 INFO [make_forcing_links] ssh 2026-03-09 forcing file links on arbutus.cloud-nowcast created
2026-03-09 09:34:00,117 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-09 09:34:00,313 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 09:34:02,739 INFO [run_NEMO] running in process 29754
2026-03-09 09:34:02,740 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:34:02,756 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-09 09:34:02,769 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:34:03,519 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-09 09:34:03,837 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/09mar26forecast_2026-03-09T093402.826040-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-09 09:34:03,858 INFO [run_NEMO] forecast NEMO run for 2026-03-09 on arbutus.cloud-nowcast started
2026-03-09 09:34:03,872 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-09 09:34:04,018 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-09 09:34:06,142 INFO [watch_NEMO] running in process 30027
2026-03-09 09:34:06,145 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:34:06,147 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-09 09:34:06,149 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:34:06,238 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-09 09:34:23,135 INFO [download_results] nowcast 2026-03-09 results files from arbutus.cloud-nowcast downloaded
2026-03-09 09:34:23,138 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-09 09:34:23,299 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 09:34:23,305 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-03-08'], host='localhost')
2026-03-09 09:34:23,310 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 09:34:25,242 INFO [make_CHS_currents_file] running in process 1212792
2026-03-09 09:34:25,243 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:34:25,244 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-09 09:34:25,245 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:34:26,242 INFO [make_plots] running in process 1212791
2026-03-09 09:34:26,243 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:34:26,244 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-09 09:34:26,245 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:34:26,247 INFO [make_plots] running in process 1212789
2026-03-09 09:34:26,248 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 09:34:26,249 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-09 09:34:26,249 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-09 09:34:26,614 INFO [make_plots] preparing render list for 2026-03-08 NEMO nowcast-blue comparison figures
2026-03-09 09:34:26,618 INFO [make_plots] preparing render list for 2026-03-09 NEMO nowcast-blue research figures
2026-03-09 09:34:26,678 INFO [make_plots] added SH_wind figure to 2026-03-08 NEMO nowcast-blue comparison render list
2026-03-09 09:34:26,680 INFO [make_plots] added Compare_VENUS_East figure to 2026-03-08 NEMO nowcast-blue comparison render list
2026-03-09 09:34:26,681 INFO [make_plots] added Compare_VENUS_Central figure to 2026-03-08 NEMO nowcast-blue comparison render list
2026-03-09 09:34:26,682 INFO [make_plots] starting to render nemo nowcast comparison 08mar26 figures
2026-03-09 09:34:27,856 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-blue research render list
2026-03-09 09:34:28,850 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-blue research render list
2026-03-09 09:34:28,850 INFO [make_plots] added Currents_sections_and_surface figure to 2026-03-09 NEMO nowcast-blue research render list
2026-03-09 09:34:28,852 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-03-09 NEMO nowcast-blue research render list
2026-03-09 09:34:28,852 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-03-09 NEMO nowcast-blue research render list
2026-03-09 09:34:28,853 INFO [make_plots] starting to render nemo nowcast research 09mar26 figures
2026-03-09 09:34:57,381 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-09 for nowcast
2026-03-09 09:34:57,384 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-09 09:35:30,142 INFO [make_plots] finished rendering nemo nowcast research 09mar26 figures
2026-03-09 09:35:30,145 INFO [make_plots] nemo research plots for 2026-03-09 nowcast completed
2026-03-09 09:35:30,162 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-09 09:35:52,091 INFO [make_plots] finished rendering nemo nowcast comparison 08mar26 figures
2026-03-09 09:35:52,094 INFO [make_plots] nemo comparison plots for 2026-03-08 nowcast completed
2026-03-09 09:35:52,099 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-09 09:39:06,236 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19088426 = 2026-03-10 05:36:40 UTC, 15.6% complete
2026-03-09 09:44:06,237 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19089000 = 2026-03-10 11:59:20 UTC, 33.3% complete
2026-03-09 09:49:06,237 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19089546 = 2026-03-10 18:03:20 UTC, 50.2% complete
2026-03-09 09:54:06,239 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19090088 = 2026-03-11 00:04:40 UTC, 66.9% complete
2026-03-09 09:59:06,240 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19090632 = 2026-03-11 06:07:20 UTC, 83.7% complete
2026-03-09 10:04:06,242 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19091160 = 2026-03-11 11:59:20 UTC, 100.0% complete
2026-03-09 10:09:06,246 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-03-09 10:09:06,250 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-09 10:09:06,593 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 10:09:06,608 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 10:09:06,613 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:09:06,621 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:09:06,626 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:09:06,640 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:09:06,642 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:09:08,424 INFO [download_results] running in process 1213215
2026-03-09 10:09:08,425 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:08,427 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-09 10:09:08,428 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:08,429 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/09mar26 to /results/SalishSea/forecast.202111
2026-03-09 10:09:08,453 INFO [upload_forcing] running in process 1213212
2026-03-09 10:09:08,455 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:08,456 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-09 10:09:08,458 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:08,468 INFO [upload_forcing] running in process 1213214
2026-03-09 10:09:08,470 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:08,472 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-09 10:09:08,474 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:08,483 INFO [upload_forcing] running in process 1213213
2026-03-09 10:09:08,485 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:08,488 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-09 10:09:08,491 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:08,494 INFO [upload_forcing] running in process 1213210
2026-03-09 10:09:08,496 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:08,499 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-09 10:09:08,501 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:09,128 INFO [upload_forcing] turbidity 2026-03-09 forcing files upload to optimum-hindcast completed
2026-03-09 10:09:09,132 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 10:09:09,440 INFO [make_ww3_wind_file] running in process 30981
2026-03-09 10:09:09,441 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:09,442 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-09 10:09:09,446 INFO [make_ww3_current_file] running in process 30980
2026-03-09 10:09:09,447 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:09,449 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-09 10:09:09,502 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:09,503 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-09 forecast run
2026-03-09 10:09:09,513 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:09,514 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-09 forecast run
2026-03-09 10:09:09,526 INFO [upload_forcing] turbidity 2026-03-09 forcing files upload to arbutus.cloud-nowcast completed
2026-03-09 10:09:09,527 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 10:09:09,702 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:09:10,711 INFO [upload_forcing] turbidity 2026-03-09 forcing files upload to orcinus-nowcast-agrif completed
2026-03-09 10:09:10,716 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 10:09:12,056 INFO [make_forcing_links] running in process 1213248
2026-03-09 10:09:12,058 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:12,060 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-09 10:09:12,061 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:13,534 INFO [upload_forcing] turbidity 2026-03-09 forcing files upload to robot.nibi completed
2026-03-09 10:09:13,537 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-09 10:09:14,790 INFO [make_forcing_links] nowcast-green 2026-03-09 forcing file links on arbutus.cloud-nowcast created
2026-03-09 10:09:14,794 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-09 10:09:14,951 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 10:09:18,930 INFO [run_NEMO] running in process 31609
2026-03-09 10:09:18,931 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:18,933 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-09 10:09:18,935 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:20,225 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-09 10:09:20,555 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/09mar26nowcast-green_2026-03-09T100918.996627-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-09 10:09:20,572 INFO [run_NEMO] nowcast-green NEMO run for 2026-03-09 on arbutus.cloud-nowcast started
2026-03-09 10:09:20,577 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-09 10:09:20,738 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-03-09 10:09:21,897 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-09 forecast run
2026-03-09 10:09:21,900 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-09 10:09:22,819 INFO [watch_NEMO] running in process 31888
2026-03-09 10:09:22,824 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:22,828 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-09 10:09:22,832 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:09:22,896 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-09 10:09:46,458 INFO [download_results] forecast 2026-03-09 results files from arbutus.cloud-nowcast downloaded
2026-03-09 10:09:46,461 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-09 10:09:46,618 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:09:49,394 INFO [make_CHS_currents_file] running in process 1213266
2026-03-09 10:09:49,395 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:09:49,397 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-09 10:09:49,398 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:10:32,542 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-09 for forecast
2026-03-09 10:10:32,546 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-09 10:10:32,712 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:10:34,193 INFO [update_forecast_datasets] running in process 1213300
2026-03-09 10:10:34,194 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:10:34,195 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-03-09 10:10:34,197 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:10:34,198 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/09mar26 run
2026-03-09 10:10:34,497 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/09mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-09 10:10:34,498 INFO [update_forecast_datasets] nemo 2026-03-09 forecast rolling forecast datasets updated
2026-03-09 10:10:34,501 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-09 10:10:34,662 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-09 10:10:34,669 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:10:34,676 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:10:35,940 INFO [ping_erddap] running in process 1213307
2026-03-09 10:10:35,941 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:10:35,942 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-09 10:10:35,943 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:10:35,954 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-09 10:10:35,956 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 10:10:38,223 INFO [make_surface_current_tiles] running in process 1213310
2026-03-09 10:10:38,225 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:10:38,226 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-03-09 10:10:38,227 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:10:38,229 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-09 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/09mar26
2026-03-09 10:10:40,311 INFO [make_plots] running in process 1213309
2026-03-09 10:10:40,312 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:10:40,313 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-09 10:10:40,315 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:10:40,729 INFO [make_plots] preparing render list for 2026-03-09 NEMO forecast publish figures
2026-03-09 10:10:41,001 INFO [make_plots] added Website_thumbnail figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,002 INFO [make_plots] added Threshold_website figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,003 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,003 INFO [make_plots] added Boundary Bay figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,004 INFO [make_plots] added Campbell River figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,005 INFO [make_plots] added Cherry Point figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,006 INFO [make_plots] added Friday Harbor figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,007 INFO [make_plots] added Halfmoon Bay figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,008 INFO [make_plots] added Nanaimo figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,009 INFO [make_plots] added Neah Bay figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,010 INFO [make_plots] added New Westminster figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,011 INFO [make_plots] added Patricia Bay figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,011 INFO [make_plots] added Point Atkinson figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,012 INFO [make_plots] added Port Renfrew figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,013 INFO [make_plots] added Sand Heads figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,014 INFO [make_plots] added Sandy Cove figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,015 INFO [make_plots] added Squamish figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,016 INFO [make_plots] added Victoria figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,016 INFO [make_plots] added Woodwards Landing figure to 2026-03-09 NEMO forecast publish render list
2026-03-09 10:10:41,017 INFO [make_plots] starting to render nemo forecast publish 09mar26 figures
2026-03-09 10:11:48,962 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-09 forecast run
2026-03-09 10:11:48,966 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-09 10:11:49,289 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 10:11:51,483 INFO [run_ww3] running in process 32031
2026-03-09 10:11:51,484 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:11:51,484 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-09 10:11:51,485 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:11:51,488 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-09T101151.482888-0700
2026-03-09 10:11:51,490 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-09T101151.482888-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-09 10:11:51,505 INFO [run_ww3] nowcast WaveWatch3 run for 2026-03-09 on arbutus.cloud-nowcast started
2026-03-09 10:11:51,509 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-09 10:11:51,822 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-09 10:11:53,855 INFO [watch_ww3] running in process 32124
2026-03-09 10:11:53,856 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:11:53,857 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-09 10:11:53,858 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:11:53,889 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-09 10:14:22,896 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19085934 = 2026-03-09 01:55:20 UTC, 8.0% complete
2026-03-09 10:16:53,892 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete
2026-03-09 10:18:14,534 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-09 10:18:14,535 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-09 10:18:36,392 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:19:22,897 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19086126 = 2026-03-09 04:03:20 UTC, 16.9% complete
2026-03-09 10:19:40,589 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-09 10:19:40,590 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-09 10:20:02,005 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:20:41,919 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:20:46,073 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/09mar26
2026-03-09 10:20:59,462 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:21:16,994 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:21:33,719 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:21:51,314 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:21:53,891 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 189, 43.8% complete
2026-03-09 10:22:07,866 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:22:24,035 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-09 10:22:24,036 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-09 10:22:40,206 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:22:58,038 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-04 00:00:00Z to 2026-03-11 11:55:00Z
2026-03-09 10:23:13,714 INFO [make_plots] finished rendering nemo forecast publish 09mar26 figures
2026-03-09 10:23:13,717 INFO [make_plots] nemo publish plots for 2026-03-09 forecast completed
2026-03-09 10:23:13,724 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-09 10:23:13,861 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:23:18,530 INFO [make_feeds] running in process 1218007
2026-03-09 10:23:18,531 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:23:18,532 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-09 10:23:18,534 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:23:33,036 INFO [make_feeds] ATOM feeds for 2026-03-09 forecast run completed
2026-03-09 10:23:33,038 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-09 10:24:22,898 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19086319 = 2026-03-09 06:12:00 UTC, 25.8% complete
2026-03-09 10:25:29,502 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/09mar26
2026-03-09 10:25:29,527 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-09 forecast into /results/nowcast-sys/figures/surface_currents/forecast/09mar26
2026-03-09 10:25:29,528 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-09 forecast completed
2026-03-09 10:25:30,013 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-09 10:26:53,892 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 291, 67.4% complete
2026-03-09 10:29:22,899 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19086513 = 2026-03-09 08:21:20 UTC, 34.8% complete
2026-03-09 10:31:53,893 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 390, 90.3% complete
2026-03-09 10:34:22,918 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19086706 = 2026-03-09 10:30:00 UTC, 43.8% complete
2026-03-09 10:36:53,894 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-09 10:36:53,899 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-09 10:36:54,370 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 10:36:54,377 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-09'], host='arbutus.cloud-nowcast')
2026-03-09 10:36:55,607 INFO [download_wwatch3_results] running in process 1218226
2026-03-09 10:36:55,608 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:36:55,610 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-03-09 10:36:55,611 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:36:56,918 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-09 10:36:56,920 INFO [run_ww3] forecast WaveWatch3 run for 2026-03-09 on arbutus.cloud-nowcast started
2026-03-09 10:36:57,229 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-09 10:36:59,900 INFO [watch_ww3] running in process 32670
2026-03-09 10:36:59,902 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 10:36:59,903 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-09 10:36:59,905 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-09 10:36:59,968 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-09 10:37:30,732 INFO [download_wwatch3_results] nowcast 2026-03-09 results files from arbutus.cloud-nowcast downloaded
2026-03-09 10:37:30,735 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-09 10:39:22,901 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19086900 = 2026-03-09 12:39:20 UTC, 52.8% complete
2026-03-09 10:41:59,971 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2026-03-09 10:44:22,905 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19087092 = 2026-03-09 14:47:20 UTC, 61.6% complete
2026-03-09 10:46:59,972 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2026-03-09 10:49:22,906 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19087285 = 2026-03-09 16:56:00 UTC, 70.6% complete
2026-03-09 10:51:59,974 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 288, 44.4% complete
2026-03-09 10:54:22,908 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19087475 = 2026-03-09 19:02:40 UTC, 79.4% complete
2026-03-09 10:56:59,976 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 387, 59.7% complete
2026-03-09 10:59:22,909 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19087668 = 2026-03-09 21:11:20 UTC, 88.3% complete
2026-03-09 11:01:59,976 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 489, 75.5% complete
2026-03-09 11:04:22,910 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19087861 = 2026-03-09 23:20:00 UTC, 97.3% complete
2026-03-09 11:06:59,977 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 594, 91.7% complete
2026-03-09 11:09:22,911 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19087920 = 2026-03-09 23:59:20 UTC, 100.0% complete
2026-03-09 11:11:59,979 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-09 11:11:59,983 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-09 11:12:00,463 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 11:12:01,737 INFO [download_wwatch3_results] running in process 1218544
2026-03-09 11:12:01,739 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:12:01,740 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-03-09 11:12:01,741 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:12:41,759 INFO [download_wwatch3_results] forecast 2026-03-09 results files from arbutus.cloud-nowcast downloaded
2026-03-09 11:12:41,763 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-09 11:12:42,061 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 11:12:43,267 INFO [update_forecast_datasets] running in process 1218559
2026-03-09 11:12:43,268 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:12:43,270 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-03-09 11:12:43,270 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:12:43,271 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/09mar26 run
2026-03-09 11:12:43,275 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/09mar26 run
2026-03-09 11:12:43,276 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/09mar26 run
2026-03-09 11:12:43,334 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/09mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-09 11:12:43,335 INFO [update_forecast_datasets] wwatch3 2026-03-09 forecast rolling forecast datasets updated
2026-03-09 11:12:43,339 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-09 11:12:43,654 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-09 11:12:44,700 INFO [ping_erddap] running in process 1218564
2026-03-09 11:12:44,701 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:12:44,702 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-09 11:12:44,703 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:12:44,705 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-09 11:12:44,707 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 11:12:45,013 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 11:12:48,588 INFO [make_plots] running in process 1218569
2026-03-09 11:12:48,590 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:12:48,591 INFO [make_plots] publishing log messages to tcp://*:5562
2026-03-09 11:12:48,592 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:12:48,593 INFO [make_plots] preparing render list for 2026-03-09 WaveWatch3 forecast publish figures
2026-03-09 11:12:48,594 INFO [make_plots] added Halibut Bank figure to 2026-03-09 WaveWatch3 forecast publish render list
2026-03-09 11:12:48,595 INFO [make_plots] added Sentry Shoal figure to 2026-03-09 WaveWatch3 forecast publish render list
2026-03-09 11:12:48,596 INFO [make_plots] starting to render wwatch3 forecast publish 09mar26 figures
2026-03-09 11:12:49,175 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-09 11:12:55,011 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-09 11:13:00,297 INFO [make_plots] finished rendering wwatch3 forecast publish 09mar26 figures
2026-03-09 11:13:00,298 INFO [make_plots] wwatch3 publish plots for 2026-03-09 forecast completed
2026-03-09 11:13:00,301 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-09 11:14:22,925 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-03-09 11:14:22,931 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-09 11:14:23,399 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 11:14:24,698 INFO [download_results] running in process 1218614
2026-03-09 11:14:24,699 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:14:24,701 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-09 11:14:24,702 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:14:24,703 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/09mar26 to /results2/SalishSea/nowcast-green.202111
2026-03-09 11:16:40,961 INFO [download_results] nowcast-green 2026-03-09 results files from arbutus.cloud-nowcast downloaded
2026-03-09 11:16:40,966 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-09 11:16:41,277 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 11:16:41,281 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-03-09 11:16:41,287 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 11:16:41,293 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 11:16:41,298 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-03-09'], host='localhost')
2026-03-09 11:16:42,606 INFO [ping_erddap] running in process 1218644
2026-03-09 11:16:42,607 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:16:42,608 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-09 11:16:42,609 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:16:42,614 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-03-09 11:16:42,616 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-09 11:16:43,916 INFO [make_averaged_dataset] running in process 1218646
2026-03-09 11:16:43,917 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:16:43,918 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2026-03-09 11:16:43,919 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:16:43,920 INFO [make_averaged_dataset] creating day-averaged dataset for 09-Mar-2026 biology
2026-03-09 11:16:43,939 INFO [make_averaged_dataset] running in process 1218645
2026-03-09 11:16:43,940 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:16:43,941 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-03-09 11:16:43,942 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:16:43,943 INFO [make_averaged_dataset] creating day-averaged dataset for 09-Mar-2026 physics
2026-03-09 11:16:43,980 INFO [make_averaged_dataset] running in process 1218647
2026-03-09 11:16:43,981 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:16:43,982 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-03-09 11:16:43,983 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:16:43,984 INFO [make_averaged_dataset] creating day-averaged dataset for 09-Mar-2026 chemistry
2026-03-09 11:16:44,833 INFO [make_plots] running in process 1218642
2026-03-09 11:16:44,834 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 11:16:44,835 INFO [make_plots] publishing log messages to tcp://*:5566
2026-03-09 11:16:44,836 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-09 11:16:45,240 INFO [make_plots] preparing render list for 2026-03-09 NEMO nowcast-green research figures
2026-03-09 11:16:47,153 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:48,009 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:48,803 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:49,611 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:50,393 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:51,171 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:51,905 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:52,760 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:53,595 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:54,433 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:55,269 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:55,978 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:57,025 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:57,026 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:57,027 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:57,028 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-03-09 NEMO nowcast-green research render list
2026-03-09 11:16:57,029 INFO [make_plots] starting to render nemo nowcast-green research 09mar26 figures
2026-03-09 11:17:35,579 INFO [make_averaged_dataset] day-averaged dataset for 09-Mar-2026 physics created
2026-03-09 11:17:35,582 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-09 11:17:41,303 INFO [make_averaged_dataset] day-averaged dataset for 09-Mar-2026 chemistry created
2026-03-09 11:17:41,306 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-09 11:17:50,206 INFO [make_averaged_dataset] day-averaged dataset for 09-Mar-2026 biology created
2026-03-09 11:17:50,209 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-09 12:16:58,195 INFO [make_plots] finished rendering nemo nowcast-green research 09mar26 figures
2026-03-09 12:16:58,208 INFO [make_plots] nemo research plots for 2026-03-09 nowcast-green completed
2026-03-09 12:16:58,272 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-09 14:45:31,764 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260309/18/
2026-03-09 14:45:31,765 INFO [collect_weather] 2026-03-09 2.5km weather forecast 18 collection complete
2026-03-09 14:45:31,782 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-09 14:45:32,460 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-03-09 14:45:32,468 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-03-09 14:45:32,473 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-03-09 14:45:32,478 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-03-10'], host='localhost')
2026-03-09 14:45:32,957 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260309/18/
2026-03-09 14:45:32,958 INFO [crop_gribs] 2026-03-09 18 GRIBs cropping complete
2026-03-09 14:45:32,963 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-09 14:45:34,340 INFO [download_weather] running in process 1221342
2026-03-09 14:45:34,342 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 14:45:34,343 INFO [download_weather] publishing log messages to tcp://*:5562
2026-03-09 14:45:34,344 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-09 14:45:34,345 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260309
2026-03-09 14:45:34,346 INFO [download_weather] running in process 1221344
2026-03-09 14:45:34,348 INFO [collect_weather] running in process 1221345
2026-03-09 14:45:34,349 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 14:45:34,351 INFO [download_weather] publishing log messages to tcp://*:5563
2026-03-09 14:45:34,352 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-09 14:45:34,353 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260309
2026-03-09 14:45:34,354 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 14:45:34,355 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-03-09 14:45:34,356 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-09 14:45:34,379 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-03-09 14:45:35,640 INFO [crop_gribs] running in process 1221346
2026-03-09 14:45:35,641 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 14:45:35,642 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-03-09 14:45:35,644 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-09 14:45:35,645 INFO [crop_gribs] cropping 2026-03-10 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-03-09 14:45:35,658 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260310/00/
2026-03-09 14:49:18,441 INFO [download_weather] 2026-03-09 1km weather forecast 00 downloads complete
2026-03-09 14:49:18,444 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-09 14:49:20,744 INFO [download_weather] 2026-03-09 1km weather forecast 12 downloads complete
2026-03-09 14:49:20,749 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-09 20:41:12,041 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260310/00/
2026-03-09 20:41:12,042 INFO [collect_weather] 2026-03-10 2.5km weather forecast 00 collection complete
2026-03-09 20:41:12,044 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-09 20:41:12,551 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-03-09 20:41:12,557 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-03-10'], host='localhost')
2026-03-09 20:41:13,749 INFO [collect_weather] running in process 1225029
2026-03-09 20:41:13,750 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 20:41:13,751 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-09 20:41:13,752 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-09 20:41:13,765 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-03-09 20:41:14,562 INFO [crop_gribs] running in process 1225031
2026-03-09 20:41:14,563 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-09 20:41:14,564 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-03-09 20:41:14,566 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-09 20:41:14,566 INFO [crop_gribs] cropping 2026-03-10 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-03-09 20:41:14,578 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260310/06/
2026-03-09 22:45:36,697 INFO [crop_gribs] crop_gribs 00 has watched for 8h; retrying remaining 1 file(s)
2026-03-09 22:45:39,473 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260310/00/
2026-03-09 22:45:39,474 INFO [crop_gribs] 2026-03-10 00 GRIBs cropping complete
2026-03-09 22:45:39,476 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-10 02:44:55,461 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260310/06/
2026-03-10 02:44:55,462 INFO [collect_weather] 2026-03-10 2.5km weather forecast 06 collection complete
2026-03-10 02:44:55,465 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-10 02:44:55,952 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:55,958 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:55,963 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:55,969 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:55,973 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:55,980 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:55,983 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:55,987 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:55,995 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:56,008 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:56,028 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:56,044 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:56,059 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:56,072 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:56,097 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-03-10 02:44:56,104 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-03-10 02:44:56,120 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-03-10 02:44:56,150 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-03-09'], host='localhost')
2026-03-10 02:44:56,175 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-03-10 02:44:56,190 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-03-10 02:44:56,199 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-03-10 02:44:58,326 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260310/06/
2026-03-10 02:44:58,326 INFO [crop_gribs] 2026-03-10 06 GRIBs cropping complete
2026-03-10 02:44:58,328 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-10 02:44:59,216 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-03-10 02:44:59,296 INFO [collect_weather] running in process 1229365
2026-03-10 02:44:59,296 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:44:59,297 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-10 02:44:59,298 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:44:59,299 INFO [collect_NeahBay_ssh] running in process 1229364
2026-03-10 02:44:59,300 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:44:59,301 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2026-03-10 02:44:59,302 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:44:59,303 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-03-10
2026-03-10 02:44:59,326 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-03-10 02:45:00,294 INFO [collect_river_data] running in process 1229348
2026-03-10 02:45:00,295 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,296 INFO [collect_river_data] publishing log messages to tcp://*:5563
2026-03-10 02:45:00,297 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,298 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-03-09
2026-03-10 02:45:00,312 INFO [collect_river_data] running in process 1229352
2026-03-10 02:45:00,313 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,314 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-10 02:45:00,314 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,315 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-03-09
2026-03-10 02:45:00,326 INFO [collect_river_data] running in process 1229351
2026-03-10 02:45:00,327 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,327 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-10 02:45:00,329 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,330 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-03-09
2026-03-10 02:45:00,350 INFO [collect_river_data] running in process 1229349
2026-03-10 02:45:00,351 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,352 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-10 02:45:00,353 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,354 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-03-09
2026-03-10 02:45:00,358 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-03-10 02:45:00,359 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-03-09 completed
2026-03-10 02:45:00,360 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:00,373 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-03-10 02:45:00,374 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-03-09 completed
2026-03-10 02:45:00,383 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-03-10 02:45:00,384 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-03-09 completed
2026-03-10 02:45:00,396 INFO [collect_river_data] running in process 1229345
2026-03-10 02:45:00,397 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,398 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-10 02:45:00,398 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,399 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-03-09
2026-03-10 02:45:00,425 INFO [collect_river_data] running in process 1229354
2026-03-10 02:45:00,426 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,428 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-03-10 02:45:00,429 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,429 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-03-09
2026-03-10 02:45:00,440 INFO [collect_river_data] running in process 1229353
2026-03-10 02:45:00,441 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,442 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-03-10 02:45:00,443 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,444 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-03-09
2026-03-10 02:45:00,445 INFO [collect_river_data] running in process 1229347
2026-03-10 02:45:00,448 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,448 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-03-10 02:45:00,449 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,450 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-03-09
2026-03-10 02:45:00,482 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-03-10 02:45:00,483 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-03-09 completed
2026-03-10 02:45:00,486 INFO [collect_river_data] running in process 1229356
2026-03-10 02:45:00,487 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,487 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-03-10 02:45:00,488 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,489 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-03-09
2026-03-10 02:45:00,491 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-03-09 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-03-10 02:45:00,491 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-03-10 02:45:00,492 INFO [collect_river_data] ECCC Capilano river data collection for 2026-03-09 completed
2026-03-10 02:45:00,492 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-03-09 completed
2026-03-10 02:45:00,495 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-03-10 02:45:00,496 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-03-09 completed
2026-03-10 02:45:00,517 INFO [collect_river_data] running in process 1229355
2026-03-10 02:45:00,518 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,519 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-03-10 02:45:00,521 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,521 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-03-09
2026-03-10 02:45:00,528 INFO [collect_river_data] running in process 1229358
2026-03-10 02:45:00,528 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,529 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-03-10 02:45:00,530 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,531 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-03-09
2026-03-10 02:45:00,533 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-03-10 02:45:00,533 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-03-09 completed
2026-03-10 02:45:00,544 INFO [collect_river_data] running in process 1229359
2026-03-10 02:45:00,545 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,546 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-03-10 02:45:00,547 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,548 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-03-09
2026-03-10 02:45:00,560 INFO [collect_river_data] running in process 1229350
2026-03-10 02:45:00,563 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,564 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-03-10 02:45:00,565 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,565 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-03-09
2026-03-10 02:45:00,566 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-03-10 02:45:00,567 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-03-09 completed
2026-03-10 02:45:00,574 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-03-10 02:45:00,575 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-03-09 completed
2026-03-10 02:45:00,576 INFO [collect_river_data] running in process 1229357
2026-03-10 02:45:00,577 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,578 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-03-10 02:45:00,579 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,579 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-03-09
2026-03-10 02:45:00,585 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-03-10 02:45:00,586 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-03-09 completed
2026-03-10 02:45:00,589 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-03-09 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-03-10 02:45:00,590 INFO [collect_river_data] ECCC Fraser river data collection for 2026-03-09 completed
2026-03-10 02:45:00,600 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-03-09 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-03-10 02:45:00,601 INFO [collect_river_data] ECCC Englishman river data collection for 2026-03-09 completed
2026-03-10 02:45:00,612 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-03-10 02:45:00,613 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-03-09 completed
2026-03-10 02:45:00,659 INFO [collect_NeahBay_ssh] 2026-03-10T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-03-10 02:45:00,828 INFO [get_vfpa_hadcp] running in process 1229363
2026-03-10 02:45:00,829 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,830 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2026-03-10 02:45:00,831 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,832 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-09
2026-03-10 02:45:00,835 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:00,982 INFO [crop_gribs] running in process 1229366
2026-03-10 02:45:00,984 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:00,985 INFO [crop_gribs] publishing log messages to tcp://*:5579
2026-03-10 02:45:00,986 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:00,987 INFO [crop_gribs] cropping 2026-03-10 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-03-10 02:45:01,001 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260310/12/
2026-03-10 02:45:01,208 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:01,630 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:01,638 INFO [get_onc_ferry] running in process 1229362
2026-03-10 02:45:01,639 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:01,640 INFO [get_onc_ferry] publishing log messages to tcp://*:5563
2026-03-10 02:45:01,641 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:01,642 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-03-09
2026-03-10 02:45:01,679 INFO [get_onc_ctd] running in process 1229361
2026-03-10 02:45:01,681 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:01,682 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2026-03-10 02:45:01,683 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:01,684 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-03-09
2026-03-10 02:45:01,763 INFO [get_onc_ctd] running in process 1229360
2026-03-10 02:45:01,764 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:01,766 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2026-03-10 02:45:01,767 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:01,768 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-03-09
2026-03-10 02:45:01,896 INFO [grib_to_netcdf] running in process 1229603
2026-03-10 02:45:01,897 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:01,898 INFO [grib_to_netcdf] publishing log messages to tcp://*:5591
2026-03-10 02:45:01,899 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:01,900 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-10 forecast2 run
2026-03-10 02:45:01,967 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:02,298 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:02,627 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:02,950 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:03,335 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:03,555 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-09 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202603.nc
2026-03-10 02:45:03,556 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-03 netcdf file
2026-03-10 02:45:03,650 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-10 02:45:03,959 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 02:45:03,966 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:04,447 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:04,789 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:05,144 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:05,552 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 02:45:05,946 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2026-03-10 02:45:06,329 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2026-03-10 02:45:07,614 INFO [ping_erddap] running in process 1229864
2026-03-10 02:45:07,616 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:07,617 INFO [ping_erddap] publishing log messages to tcp://*:5564
2026-03-10 02:45:07,618 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:07,619 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2026-03-10 02:45:07,621 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 02:45:07,905 INFO [make_ssh_files] running in process 1229845
2026-03-10 02:45:07,906 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:07,907 INFO [make_ssh_files] publishing log messages to tcp://*:5566
2026-03-10 02:45:07,909 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:07,910 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-03-10 NOAA Neah Bay 00Z observation and forecast values
2026-03-10 02:45:10,000 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d05.nc
2026-03-10 02:45:10,203 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d06.nc
2026-03-10 02:45:10,391 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d07.nc
2026-03-10 02:45:10,606 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d08.nc
2026-03-10 02:45:10,793 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d09.nc
2026-03-10 02:45:11,037 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d10.nc
2026-03-10 02:45:11,258 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d11.nc
2026-03-10 02:45:11,496 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d12.nc
2026-03-10 02:45:11,679 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d13.nc
2026-03-10 02:45:11,820 INFO [make_ssh_files] sea surface height boundary file for 2026-03-10 forecast2 run created
2026-03-10 02:45:11,824 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-10 02:45:12,225 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-10 02:45:12,229 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-10 02:45:13,718 INFO [make_201702_runoff_file] running in process 1229917
2026-03-10 02:45:13,719 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:13,720 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2026-03-10 02:45:13,721 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:13,745 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-10 02:45:13,840 INFO [make_201702_runoff_file] 2026-03-10 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-10 02:45:13,842 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-10 02:45:14,127 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d11.nc
2026-03-10 02:45:14,424 INFO [make_runoff_file] running in process 1229915
2026-03-10 02:45:14,426 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:14,427 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2026-03-10 02:45:14,427 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:14,431 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-09
2026-03-10 02:45:15,459 ERROR [make_runoff_file] no 2026-03-09 discharge obs for SkagitMountVernon: patching
2026-03-10 02:45:16,112 ERROR [make_runoff_file] no 2026-03-09 discharge obs for SnohomishMonroe: patching
2026-03-10 02:45:16,359 ERROR [make_runoff_file] no 2026-03-09 discharge obs for NisquallyMcKenna: patching
2026-03-10 02:45:16,484 ERROR [make_runoff_file] no 2026-03-09 discharge obs for GreenwaterGreenwater: patching
2026-03-10 02:45:17,242 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d12.nc
2026-03-10 02:45:17,248 INFO [grib_to_netcdf] 2026-03-10 NEMO-atmos forcing file for forecast2 created
2026-03-10 02:45:17,251 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-10 02:45:17,857 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-09: /results/forcing/rivers/R202108Dailies_y2026m03d09.nc
2026-03-10 02:45:17,858 INFO [make_runoff_file] 2026-03-09 runoff file creation completed
2026-03-10 02:45:17,860 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-10 02:45:18,215 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-03-10 02:45:18,220 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-03-10 02:45:18,225 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-03-10 02:45:18,236 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-03-10 02:45:19,747 INFO [upload_forcing] running in process 1229961
2026-03-10 02:45:19,751 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:19,752 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-10 02:45:19,753 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:19,773 INFO [upload_forcing] running in process 1229960
2026-03-10 02:45:19,774 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:19,775 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-10 02:45:19,776 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:19,793 INFO [upload_forcing] running in process 1229962
2026-03-10 02:45:19,794 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:19,795 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-10 02:45:19,796 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:19,800 INFO [upload_forcing] running in process 1229959
2026-03-10 02:45:19,801 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:19,802 INFO [upload_forcing] publishing log messages to tcp://*:5568
2026-03-10 02:45:19,803 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:20,514 INFO [get_onc_ctd] 2026-03-09 ONC SCVIP CTD T&S file created
2026-03-10 02:45:20,516 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-10 02:45:20,833 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-03-10 02:45:21,134 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d09.nc
2026-03-10 02:45:21,213 INFO [upload_forcing] forecast2 2026-03-10 forcing files upload to optimum-hindcast completed
2026-03-10 02:45:21,216 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 02:45:21,233 INFO [get_onc_ctd] 2026-03-09 ONC SEVIP CTD T&S file created
2026-03-10 02:45:21,582 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-10 02:45:21,947 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-03-10 02:45:22,119 INFO [ping_erddap] running in process 1229989
2026-03-10 02:45:22,120 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:22,121 INFO [ping_erddap] publishing log messages to tcp://*:5567
2026-03-10 02:45:22,122 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:22,124 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-03-10 02:45:22,125 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 02:45:22,684 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-03-09
2026-03-10 02:45:22,741 INFO [upload_forcing] forecast2 2026-03-10 forcing files upload to arbutus.cloud-nowcast completed
2026-03-10 02:45:22,746 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 02:45:23,067 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 02:45:23,154 INFO [ping_erddap] running in process 1230000
2026-03-10 02:45:23,155 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:23,156 INFO [ping_erddap] publishing log messages to tcp://*:5565
2026-03-10 02:45:23,157 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:23,158 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-03-10 02:45:23,159 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 02:45:24,266 INFO [make_forcing_links] running in process 1230009
2026-03-10 02:45:24,267 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:24,268 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2026-03-10 02:45:24,269 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:25,396 INFO [make_forcing_links] forecast2 2026-03-10 forcing file links on arbutus.cloud-nowcast created
2026-03-10 02:45:25,400 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-10 02:45:25,745 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 02:45:26,518 INFO [upload_forcing] forecast2 2026-03-10 forcing files upload to orcinus-nowcast-agrif completed
2026-03-10 02:45:26,521 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 02:45:27,454 INFO [upload_forcing] forecast2 2026-03-10 forcing files upload to robot.nibi completed
2026-03-10 02:45:27,458 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 02:45:27,794 INFO [run_NEMO] running in process 8390
2026-03-10 02:45:27,795 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:27,796 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-10 02:45:27,857 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:28,692 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-10 02:45:29,000 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/09mar26forecast2_2026-03-10T024527.905231-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-10 02:45:29,014 INFO [run_NEMO] forecast2 NEMO run for 2026-03-10 on arbutus.cloud-nowcast started
2026-03-10 02:45:29,018 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-10 02:45:29,362 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-10 02:45:31,369 INFO [watch_NEMO] running in process 8663
2026-03-10 02:45:31,370 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:45:31,371 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-10 02:45:31,373 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:45:31,436 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-10 02:45:45,872 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-03-09
2026-03-10 02:46:08,934 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-03-09
2026-03-10 02:46:31,124 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-03-09
2026-03-10 02:46:31,189 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,194 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,195 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,197 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-03-09
2026-03-10 02:46:31,256 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,262 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,264 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,265 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-03-09
2026-03-10 02:46:31,324 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,328 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,329 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-03-09
2026-03-10 02:46:31,395 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,399 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,400 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-03-09
2026-03-10 02:46:31,462 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,465 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-09; substituting empty dataset
2026-03-10 02:46:31,698 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-03-09 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-10 02:46:31,717 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-03-09 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-10 02:46:31,735 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-03-09 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-10 02:46:31,753 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-03-09 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-10 02:46:31,771 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-03-09 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-10 02:46:31,791 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-03-09 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-10 02:46:31,807 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-09 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-10 02:47:08,344 INFO [get_onc_ferry] 2026-03-09 ONC TWDP ferry data file created
2026-03-10 02:47:08,347 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-03-10 02:47:08,624 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-03-10 02:47:09,873 INFO [ping_erddap] running in process 1230047
2026-03-10 02:47:09,874 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 02:47:09,875 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-10 02:47:09,876 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 02:47:09,878 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-03-10 02:47:09,881 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 02:50:31,439 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19090574 = 2026-03-11 05:28:40 UTC, 18.3% complete
2026-03-10 02:55:31,441 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19091094 = 2026-03-11 11:15:20 UTC, 37.5% complete
2026-03-10 03:00:31,442 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19091647 = 2026-03-11 17:24:00 UTC, 58.0% complete
2026-03-10 03:05:31,443 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19092211 = 2026-03-11 23:40:00 UTC, 78.9% complete
2026-03-10 03:10:31,449 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19092738 = 2026-03-12 05:31:20 UTC, 98.4% complete
2026-03-10 03:15:31,433 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-03-10 03:15:31,437 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-10 03:15:32,003 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 03:15:32,016 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 03:15:32,021 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-10 03:15:33,277 INFO [download_results] running in process 1230390
2026-03-10 03:15:33,278 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:15:33,280 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-10 03:15:33,280 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:15:33,281 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/09mar26 to /results/SalishSea/forecast2.202111
2026-03-10 03:15:34,970 INFO [make_ww3_wind_file] running in process 9065
2026-03-10 03:15:34,971 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:15:34,972 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-10 03:15:34,973 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:15:34,974 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-10 forecast2 run
2026-03-10 03:15:35,065 INFO [make_ww3_current_file] running in process 9063
2026-03-10 03:15:35,066 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:15:35,067 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-10 03:15:35,069 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:15:35,070 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-10 forecast2 run
2026-03-10 03:15:43,749 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-10 forecast2 run
2026-03-10 03:15:43,753 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-10 03:17:05,739 INFO [download_results] forecast2 2026-03-09 results files from arbutus.cloud-nowcast downloaded
2026-03-10 03:17:05,744 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-10 03:17:06,097 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-10 03:17:08,999 INFO [make_CHS_currents_file] running in process 1230404
2026-03-10 03:17:09,000 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:17:09,001 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-10 03:17:09,002 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:17:49,909 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-09 for forecast2
2026-03-10 03:17:49,912 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-10 03:17:50,242 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-10 03:17:52,577 INFO [update_forecast_datasets] running in process 1230442
2026-03-10 03:17:52,578 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:17:52,580 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-10 03:17:52,743 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:17:52,744 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/09mar26 run
2026-03-10 03:19:11,914 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-10 forecast2 run
2026-03-10 03:19:11,919 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-10 03:19:12,277 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 03:19:14,445 INFO [run_ww3] running in process 9652
2026-03-10 03:19:14,446 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:19:14,448 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-10 03:19:14,449 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:19:14,453 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-10T031914.444003-0700
2026-03-10 03:19:14,454 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-10T031914.444003-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-10 03:19:14,465 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-03-10 on arbutus.cloud-nowcast started
2026-03-10 03:19:14,469 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-10 03:19:14,759 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-10 03:19:16,706 INFO [watch_ww3] running in process 9739
2026-03-10 03:19:16,707 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:19:16,709 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-10 03:19:16,710 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:19:16,740 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-10 03:22:28,892 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/09mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-10 03:22:28,893 INFO [update_forecast_datasets] nemo 2026-03-09 forecast2 rolling forecast datasets updated
2026-03-10 03:22:28,895 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-10 03:22:29,238 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-10 03:22:29,245 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-03-09'], host='localhost')
2026-03-10 03:22:29,251 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-10 03:22:30,746 INFO [ping_erddap] running in process 1230548
2026-03-10 03:22:30,748 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:22:30,748 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-10 03:22:30,749 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:22:30,758 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-10 03:22:30,759 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 03:22:32,535 INFO [make_surface_current_tiles] running in process 1230551
2026-03-10 03:22:32,537 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:22:32,538 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-10 03:22:32,540 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:22:32,542 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-09 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/09mar26
2026-03-10 03:22:34,740 INFO [make_plots] running in process 1230550
2026-03-10 03:22:34,740 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:22:34,741 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-10 03:22:34,742 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:22:35,227 INFO [make_plots] preparing render list for 2026-03-09 NEMO forecast2 publish figures
2026-03-10 03:22:35,857 INFO [make_plots] added Website_thumbnail figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,859 INFO [make_plots] added Threshold_website figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,862 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,865 INFO [make_plots] added Boundary Bay figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,866 INFO [make_plots] added Campbell River figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,869 INFO [make_plots] added Cherry Point figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,872 INFO [make_plots] added Friday Harbor figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,875 INFO [make_plots] added Halfmoon Bay figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,876 INFO [make_plots] added Nanaimo figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,879 INFO [make_plots] added Neah Bay figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,881 INFO [make_plots] added New Westminster figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,884 INFO [make_plots] added Patricia Bay figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,888 INFO [make_plots] added Point Atkinson figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,890 INFO [make_plots] added Port Renfrew figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,892 INFO [make_plots] added Sand Heads figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,894 INFO [make_plots] added Sandy Cove figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,897 INFO [make_plots] added Squamish figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,899 INFO [make_plots] added Victoria figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,902 INFO [make_plots] added Woodwards Landing figure to 2026-03-09 NEMO forecast2 publish render list
2026-03-10 03:22:35,905 INFO [make_plots] starting to render nemo forecast2 publish 09mar26 figures
2026-03-10 03:24:16,738 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2026-03-10 03:29:16,737 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 198, 36.7% complete
2026-03-10 03:31:52,548 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-10 03:31:52,549 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-10 03:32:18,354 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:33:32,451 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-10 03:33:32,452 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-10 03:33:55,295 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:34:16,738 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 291, 53.9% complete
2026-03-10 03:34:38,592 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/09mar26
2026-03-10 03:34:44,045 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:35:01,591 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:35:20,348 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:35:38,584 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:35:55,308 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:36:12,981 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:36:30,022 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-10 03:36:30,023 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-10 03:36:45,820 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:37:02,109 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-05 00:00:00Z to 2026-03-12 05:55:00Z
2026-03-10 03:37:17,753 INFO [make_plots] finished rendering nemo forecast2 publish 09mar26 figures
2026-03-10 03:37:17,755 INFO [make_plots] nemo publish plots for 2026-03-09 forecast2 completed
2026-03-10 03:37:17,763 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-10 03:37:18,093 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-03-09'], host='localhost')
2026-03-10 03:37:22,925 INFO [make_feeds] running in process 1235570
2026-03-10 03:37:22,926 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:37:22,927 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-10 03:37:22,928 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:37:39,369 INFO [make_feeds] ATOM feeds for 2026-03-09 forecast2 run completed
2026-03-10 03:37:39,372 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-10 03:37:39,696 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-03-10 03:37:40,821 INFO [clear_checklist] running in process 1235597
2026-03-10 03:37:40,822 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:37:40,823 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-03-10 03:37:40,824 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:37:40,825 INFO [clear_checklist] requesting that manager clear system state checklist
2026-03-10 03:37:40,826 INFO [manager] writing checklist to log file
2026-03-10 03:37:40,876 INFO [manager] checklist cleared
2026-03-10 03:37:40,878 INFO [clear_checklist] nowcast system checklist cleared
2026-03-10 03:37:40,881 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-03-10 03:37:41,994 INFO [rotate_logs] running in process 1235602
2026-03-10 03:37:41,995 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:37:41,996 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-03-10 03:37:41,997 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:37:41,998 INFO [rotate_logs] rotating log files