2026-03-14 03:30:03,572 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-03-14 03:30:03,581 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-03-14 03:30:03,587 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-03-14 03:30:03,588 INFO [rotate_logs] log files rotated
2026-03-14 03:30:03,598 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-03-14 03:32:25,754 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13mar26
2026-03-14 03:32:25,777 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-13 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/13mar26
2026-03-14 03:32:25,779 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-13 forecast2 completed
2026-03-14 03:32:26,251 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-14 03:34:47,746 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 399, 73.9% complete
2026-03-14 03:39:47,747 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 504, 93.3% complete
2026-03-14 03:44:47,750 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-14 03:44:47,753 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-14 03:44:48,058 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 03:44:49,385 INFO [download_wwatch3_results] running in process 1358076
2026-03-14 03:44:49,386 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:44:49,388 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-14 03:44:49,389 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:45:23,257 INFO [download_wwatch3_results] forecast2 2026-03-14 results files from arbutus.cloud-nowcast downloaded
2026-03-14 03:45:23,260 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-14 03:45:23,408 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 03:45:24,436 INFO [update_forecast_datasets] running in process 1358090
2026-03-14 03:45:24,438 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:45:24,439 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-14 03:45:24,440 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:45:24,441 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/14mar26 run
2026-03-14 03:45:24,454 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/14mar26 run
2026-03-14 03:45:24,455 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/14mar26 run
2026-03-14 03:45:57,102 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/14mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-14 03:45:57,103 INFO [update_forecast_datasets] wwatch3 2026-03-14 forecast2 rolling forecast datasets updated
2026-03-14 03:45:57,106 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-14 03:45:57,260 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-14 03:45:58,575 INFO [ping_erddap] running in process 1358105
2026-03-14 03:45:58,577 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:45:58,578 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-14 03:45:58,579 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:45:58,581 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-14 03:45:58,583 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 03:45:58,729 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 03:46:02,078 INFO [make_plots] running in process 1358111
2026-03-14 03:46:02,079 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 03:46:02,080 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-14 03:46:02,082 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-14 03:46:02,083 INFO [make_plots] preparing render list for 2026-03-14 WaveWatch3 forecast2 publish figures
2026-03-14 03:46:02,084 INFO [make_plots] added Halibut Bank figure to 2026-03-14 WaveWatch3 forecast2 publish render list
2026-03-14 03:46:02,085 INFO [make_plots] added Sentry Shoal figure to 2026-03-14 WaveWatch3 forecast2 publish render list
2026-03-14 03:46:02,086 INFO [make_plots] starting to render wwatch3 forecast2 publish 14mar26 figures
2026-03-14 03:46:02,881 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-14 03:46:08,778 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-14 03:46:13,782 INFO [make_plots] finished rendering wwatch3 forecast2 publish 14mar26 figures
2026-03-14 03:46:13,783 INFO [make_plots] wwatch3 publish plots for 2026-03-14 forecast2 completed
2026-03-14 03:46:13,787 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-14 08:40:51,989 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260314/12/
2026-03-14 08:40:51,990 INFO [collect_weather] 2026-03-14 2.5km weather forecast 12 collection complete
2026-03-14 08:40:51,993 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-14 08:40:52,374 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 08:40:52,382 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 08:40:52,388 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 08:40:52,395 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-03-13'], host='localhost')
2026-03-14 08:40:52,400 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-03-14 08:40:52,406 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-03-14 08:40:52,415 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-03-14 08:40:52,419 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-03-14 08:40:52,423 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-03-14 08:40:52,833 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260314/12/
2026-03-14 08:40:52,834 INFO [crop_gribs] 2026-03-14 12 GRIBs cropping complete
2026-03-14 08:40:52,837 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-14 08:40:53,377 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-03-14 08:40:54,538 INFO [collect_weather] running in process 1362688
2026-03-14 08:40:54,539 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:54,542 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-14 08:40:54,543 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:54,544 INFO [collect_NeahBay_ssh] running in process 1362686
2026-03-14 08:40:54,544 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:54,545 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-03-14 08:40:54,546 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:54,547 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-03-14
2026-03-14 08:40:54,559 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-03-14 08:40:54,616 INFO [download_live_ocean] running in process 1362687
2026-03-14 08:40:54,617 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:54,617 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2026-03-14 08:40:54,618 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:54,619 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-03-14
2026-03-14 08:40:55,156 INFO [download_live_ocean] downloaded 10217704 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.03.14/ubc.nc to /results/forcing/LiveOcean/downloaded/20260314/low_passed_UBC.nc
2026-03-14 08:40:55,157 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-03-14 08:40:55,235 INFO [collect_river_data] running in process 1362684
2026-03-14 08:40:55,237 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:55,237 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-14 08:40:55,238 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:55,239 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-03-13
2026-03-14 08:40:55,242 INFO [collect_river_data] running in process 1362683
2026-03-14 08:40:55,244 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:55,245 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-14 08:40:55,246 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:55,247 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-03-13
2026-03-14 08:40:55,345 INFO [collect_river_data] running in process 1362680
2026-03-14 08:40:55,346 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:55,347 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-14 08:40:55,348 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:55,349 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-03-13
2026-03-14 08:40:55,359 INFO [collect_river_data] running in process 1362682
2026-03-14 08:40:55,360 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:55,362 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-14 08:40:55,362 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:55,363 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-03-13
2026-03-14 08:40:55,503 INFO [make_turbidity_file] running in process 1362685
2026-03-14 08:40:55,504 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:55,505 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-03-14 08:40:55,506 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:55,507 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-03-14
2026-03-14 08:40:55,667 INFO [grib_to_netcdf] running in process 1362762
2026-03-14 08:40:55,668 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:55,669 INFO [grib_to_netcdf] publishing log messages to tcp://*:5570
2026-03-14 08:40:55,670 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:55,671 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-14 nowcast and forecast runs
2026-03-14 08:40:55,681 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-03-13&endDT=2026-03-13 "HTTP/1.1 200 "
2026-03-14 08:40:55,685 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-03-13&endDT=2026-03-13 "HTTP/1.1 200 "
2026-03-14 08:40:55,741 INFO [collect_NeahBay_ssh] 2026-03-14T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-03-14 08:40:55,744 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-14 08:40:55,827 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-03-14 08:40:55,828 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-03-13 completed
2026-03-14 08:40:55,867 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-03-13&endDT=2026-03-13 "HTTP/1.1 200 "
2026-03-14 08:40:55,872 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-03-13&endDT=2026-03-13 "HTTP/1.1 200 "
2026-03-14 08:40:55,876 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-03-14 08:40:55,877 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-03-13 completed
2026-03-14 08:40:55,887 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 08:40:55,893 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 08:40:55,894 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-03-14 08:40:55,895 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-03-13 completed
2026-03-14 08:40:55,900 INFO [crop_gribs] running in process 1362689
2026-03-14 08:40:55,901 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:55,902 INFO [crop_gribs] publishing log messages to tcp://*:5571
2026-03-14 08:40:55,903 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:55,904 INFO [crop_gribs] cropping 2026-03-14 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-03-14 08:40:55,918 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260314/18/
2026-03-14 08:40:55,946 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-03-13 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-03-14 08:40:55,947 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-03-13 completed
2026-03-14 08:40:56,118 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 08:40:56,207 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260314/low_passed_UBC.nc
2026-03-14 08:40:56,207 INFO [download_live_ocean] 2026-03-14 Live Ocean file for Salish Sea western boundary downloaded
2026-03-14 08:40:56,323 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 08:40:56,477 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-14 08:40:56,709 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-03-14 08:40:57,008 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-03-14'], host='localhost')
2026-03-14 08:40:58,220 INFO [make_turbidity_file] 2026-03-14 Fraser River turbidity file creation complete
2026-03-14 08:40:58,222 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-03-14 08:40:59,545 INFO [make_ssh_files] running in process 1362825
2026-03-14 08:40:59,546 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:59,547 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-03-14 08:40:59,549 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:59,550 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-03-14 NOAA Neah Bay 06Z observation and forecast values
2026-03-14 08:40:59,551 INFO [make_live_ocean_files] running in process 1362845
2026-03-14 08:40:59,552 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:40:59,553 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2026-03-14 08:40:59,554 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:40:59,554 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-03-14 Live Ocean run
2026-03-14 08:41:01,433 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d10.nc
2026-03-14 08:41:01,615 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d11.nc
2026-03-14 08:41:01,798 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d12.nc
2026-03-14 08:41:01,982 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d13.nc
2026-03-14 08:41:02,206 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d14.nc
2026-03-14 08:41:02,406 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d15.nc
2026-03-14 08:41:02,597 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d16.nc
2026-03-14 08:41:02,808 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d17.nc
2026-03-14 08:41:02,942 INFO [make_ssh_files] sea surface height boundary file for 2026-03-14 nowcast run created
2026-03-14 08:41:02,946 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-14 08:41:03,081 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-14 08:41:03,085 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-14 08:41:04,548 INFO [make_201702_runoff_file] running in process 1362912
2026-03-14 08:41:04,549 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:04,550 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2026-03-14 08:41:04,551 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:04,580 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-14 08:41:04,718 INFO [make_201702_runoff_file] 2026-03-14 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-14 08:41:04,721 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-14 08:41:05,119 INFO [make_runoff_file] running in process 1362910
2026-03-14 08:41:05,120 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:05,121 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2026-03-14 08:41:05,122 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:05,124 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-13
2026-03-14 08:41:08,020 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-13: /results/forcing/rivers/R202108Dailies_y2026m03d13.nc
2026-03-14 08:41:08,021 INFO [make_runoff_file] 2026-03-13 runoff file creation completed
2026-03-14 08:41:08,023 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-14 08:41:08,757 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m03d14.nc
2026-03-14 08:41:17,569 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d15.nc
2026-03-14 08:41:22,626 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d16.nc
2026-03-14 08:41:22,642 INFO [grib_to_netcdf] 2026-03-14 NEMO-atmos forcing file for nowcast+ created
2026-03-14 08:41:22,646 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-14 08:41:42,686 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d14.nc
2026-03-14 08:41:42,687 INFO [make_live_ocean_files] 2026-03-14 Live Ocean western boundary conditions files created
2026-03-14 08:41:42,689 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-03-14 08:41:43,005 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-03-14 08:41:43,006 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-03-14 08:41:43,007 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-03-14 08:41:43,009 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-03-14 08:41:43,010 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-03-14 08:41:44,127 INFO [ping_erddap] running in process 1362988
2026-03-14 08:41:44,128 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:44,129 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-14 08:41:44,131 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:44,132 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-03-14 08:41:44,134 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 08:41:44,270 INFO [upload_forcing] running in process 1362993
2026-03-14 08:41:44,271 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:44,272 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-14 08:41:44,273 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:44,274 INFO [upload_forcing] running in process 1362990
2026-03-14 08:41:44,276 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:44,277 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-14 08:41:44,278 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:44,290 INFO [upload_forcing] running in process 1362992
2026-03-14 08:41:44,291 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:44,292 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-14 08:41:44,293 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:44,368 INFO [upload_forcing] running in process 1362991
2026-03-14 08:41:44,369 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:44,371 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-14 08:41:44,372 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:46,032 INFO [upload_forcing] nowcast+ 2026-03-14 forcing files upload to optimum-hindcast completed
2026-03-14 08:41:46,036 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 08:41:51,477 INFO [upload_forcing] nowcast+ 2026-03-14 forcing files upload to arbutus.cloud-nowcast completed
2026-03-14 08:41:51,483 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 08:41:51,648 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 08:41:52,064 INFO [upload_forcing] nowcast+ 2026-03-14 forcing files upload to orcinus-nowcast-agrif completed
2026-03-14 08:41:52,067 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 08:41:53,114 INFO [make_forcing_links] running in process 1363025
2026-03-14 08:41:53,115 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:53,117 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-03-14 08:41:53,118 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:54,276 INFO [make_forcing_links] nowcast+ 2026-03-14 forcing file links on arbutus.cloud-nowcast created
2026-03-14 08:41:54,280 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-14 08:41:54,444 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 08:41:54,916 INFO [upload_forcing] nowcast+ 2026-03-14 forcing files upload to robot.nibi completed
2026-03-14 08:41:54,921 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 08:41:56,957 INFO [run_NEMO] running in process 17524
2026-03-14 08:41:56,959 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:41:56,960 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-14 08:41:56,971 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:41:57,707 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-14 08:41:58,016 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/14mar26nowcast_2026-03-14T084157.006255-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-14 08:41:58,030 INFO [run_NEMO] nowcast NEMO run for 2026-03-14 on arbutus.cloud-nowcast started
2026-03-14 08:41:58,034 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-14 08:41:58,191 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-14 08:42:00,169 INFO [watch_NEMO] running in process 17799
2026-03-14 08:42:00,171 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 08:42:00,172 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-14 08:42:00,173 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-14 08:42:00,255 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-14 08:47:00,257 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19097039 = 2026-03-14 05:18:40 UTC, 22.1% complete
2026-03-14 08:52:00,259 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19097585 = 2026-03-14 11:22:40 UTC, 47.4% complete
2026-03-14 08:57:00,260 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19098140 = 2026-03-14 17:32:40 UTC, 73.1% complete
2026-03-14 09:02:00,260 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19098664 = 2026-03-14 23:22:00 UTC, 97.4% complete
2026-03-14 09:07:00,264 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-03-14 09:07:00,268 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-14 09:07:00,578 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:07:00,585 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:07:01,821 INFO [download_results] running in process 1363327
2026-03-14 09:07:01,822 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:07:01,824 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-14 09:07:01,825 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:07:01,826 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/14mar26 to /results/SalishSea/nowcast-blue.202111
2026-03-14 09:07:01,931 INFO [make_forcing_links] running in process 1363326
2026-03-14 09:07:01,933 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:07:01,934 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-14 09:07:01,935 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:07:02,931 INFO [make_forcing_links] ssh 2026-03-14 forcing file links on arbutus.cloud-nowcast created
2026-03-14 09:07:02,935 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-14 09:07:03,102 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 09:07:05,213 INFO [run_NEMO] running in process 18333
2026-03-14 09:07:05,214 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:07:05,215 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-14 09:07:05,217 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:07:05,976 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-14 09:07:06,319 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/14mar26forecast_2026-03-14T090705.281524-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-14 09:07:06,334 INFO [run_NEMO] forecast NEMO run for 2026-03-14 on arbutus.cloud-nowcast started
2026-03-14 09:07:06,338 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-14 09:07:06,497 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-14 09:07:08,521 INFO [watch_NEMO] running in process 18608
2026-03-14 09:07:08,528 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:07:08,531 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-14 09:07:08,534 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:07:08,606 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-14 09:07:28,916 INFO [download_results] nowcast 2026-03-14 results files from arbutus.cloud-nowcast downloaded
2026-03-14 09:07:28,920 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-14 09:07:29,074 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:07:29,079 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-03-13'], host='localhost')
2026-03-14 09:07:29,083 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:07:31,992 INFO [make_CHS_currents_file] running in process 1363373
2026-03-14 09:07:31,993 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:07:31,994 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-14 09:07:31,995 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:07:34,056 INFO [make_plots] running in process 1363370
2026-03-14 09:07:34,057 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:07:34,058 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-14 09:07:34,059 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:07:34,061 INFO [make_plots] running in process 1363372
2026-03-14 09:07:34,062 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:07:34,063 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-14 09:07:34,064 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:07:34,426 INFO [make_plots] preparing render list for 2026-03-13 NEMO nowcast-blue comparison figures
2026-03-14 09:07:34,439 INFO [make_plots] preparing render list for 2026-03-14 NEMO nowcast-blue research figures
2026-03-14 09:07:34,486 INFO [make_plots] added SH_wind figure to 2026-03-13 NEMO nowcast-blue comparison render list
2026-03-14 09:07:34,487 INFO [make_plots] added Compare_VENUS_East figure to 2026-03-13 NEMO nowcast-blue comparison render list
2026-03-14 09:07:34,488 INFO [make_plots] added Compare_VENUS_Central figure to 2026-03-13 NEMO nowcast-blue comparison render list
2026-03-14 09:07:34,489 INFO [make_plots] starting to render nemo nowcast comparison 13mar26 figures
2026-03-14 09:07:35,788 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-blue research render list
2026-03-14 09:07:36,870 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-blue research render list
2026-03-14 09:07:36,871 INFO [make_plots] added Currents_sections_and_surface figure to 2026-03-14 NEMO nowcast-blue research render list
2026-03-14 09:07:36,872 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-03-14 NEMO nowcast-blue research render list
2026-03-14 09:07:36,873 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-03-14 NEMO nowcast-blue research render list
2026-03-14 09:07:36,874 INFO [make_plots] starting to render nemo nowcast research 14mar26 figures
2026-03-14 09:08:07,965 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-14 for nowcast
2026-03-14 09:08:07,968 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-14 09:08:41,051 INFO [make_plots] finished rendering nemo nowcast research 14mar26 figures
2026-03-14 09:08:41,054 INFO [make_plots] nemo research plots for 2026-03-14 nowcast completed
2026-03-14 09:08:41,072 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-14 09:09:00,129 INFO [make_plots] finished rendering nemo nowcast comparison 13mar26 figures
2026-03-14 09:09:00,131 INFO [make_plots] nemo comparison plots for 2026-03-13 nowcast completed
2026-03-14 09:09:00,136 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-14 09:12:08,605 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19099202 = 2026-03-15 05:20:40 UTC, 14.9% complete
2026-03-14 09:17:08,606 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19099722 = 2026-03-15 11:07:20 UTC, 30.9% complete
2026-03-14 09:22:08,607 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19100258 = 2026-03-15 17:04:40 UTC, 47.5% complete
2026-03-14 09:27:08,608 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19100784 = 2026-03-15 22:55:20 UTC, 63.7% complete
2026-03-14 09:32:08,611 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19101329 = 2026-03-16 04:58:40 UTC, 80.5% complete
2026-03-14 09:37:08,614 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19101878 = 2026-03-16 11:04:40 UTC, 97.5% complete
2026-03-14 09:42:08,619 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-03-14 09:42:08,623 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-14 09:42:09,005 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 09:42:09,020 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 09:42:09,025 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:42:09,034 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:42:09,037 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:42:09,043 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:42:09,048 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:42:10,754 INFO [download_results] running in process 1363861
2026-03-14 09:42:10,756 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:10,757 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-14 09:42:10,758 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:10,759 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/14mar26 to /results/SalishSea/forecast.202111
2026-03-14 09:42:10,837 INFO [upload_forcing] running in process 1363858
2026-03-14 09:42:10,839 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:10,840 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-14 09:42:10,841 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:10,857 INFO [upload_forcing] running in process 1363859
2026-03-14 09:42:10,859 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:10,860 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-14 09:42:10,861 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:10,872 INFO [upload_forcing] running in process 1363857
2026-03-14 09:42:10,873 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:10,874 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-14 09:42:10,875 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:10,885 INFO [upload_forcing] running in process 1363860
2026-03-14 09:42:10,887 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:10,888 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-14 09:42:10,889 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:11,564 INFO [upload_forcing] turbidity 2026-03-14 forcing files upload to optimum-hindcast completed
2026-03-14 09:42:11,568 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 09:42:11,815 INFO [make_ww3_current_file] running in process 19291
2026-03-14 09:42:11,816 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:11,817 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-14 09:42:11,831 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:11,832 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-14 forecast run
2026-03-14 09:42:11,848 INFO [make_ww3_wind_file] running in process 19317
2026-03-14 09:42:11,849 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:11,851 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-14 09:42:11,852 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:11,855 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-14 forecast run
2026-03-14 09:42:11,921 INFO [upload_forcing] turbidity 2026-03-14 forcing files upload to arbutus.cloud-nowcast completed
2026-03-14 09:42:11,926 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 09:42:12,093 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:42:13,094 INFO [upload_forcing] turbidity 2026-03-14 forcing files upload to orcinus-nowcast-agrif completed
2026-03-14 09:42:13,099 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 09:42:14,436 INFO [make_forcing_links] running in process 1363894
2026-03-14 09:42:14,438 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:14,439 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-14 09:42:14,441 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:15,553 INFO [upload_forcing] turbidity 2026-03-14 forcing files upload to robot.nibi completed
2026-03-14 09:42:15,557 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-14 09:42:15,660 INFO [make_forcing_links] nowcast-green 2026-03-14 forcing file links on arbutus.cloud-nowcast created
2026-03-14 09:42:15,717 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-14 09:42:15,879 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 09:42:20,619 INFO [run_NEMO] running in process 19943
2026-03-14 09:42:20,621 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:20,623 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-14 09:42:20,625 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:23,034 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-14 09:42:23,381 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/14mar26nowcast-green_2026-03-14T094220.724023-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-14 09:42:23,403 INFO [run_NEMO] nowcast-green NEMO run for 2026-03-14 on arbutus.cloud-nowcast started
2026-03-14 09:42:23,415 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-14 09:42:23,587 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-03-14 09:42:24,484 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-14 forecast run
2026-03-14 09:42:24,488 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-14 09:42:25,819 INFO [watch_NEMO] running in process 20226
2026-03-14 09:42:25,820 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:25,821 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-14 09:42:25,822 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:42:25,887 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-14 09:42:51,006 INFO [download_results] forecast 2026-03-14 results files from arbutus.cloud-nowcast downloaded
2026-03-14 09:42:51,011 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-14 09:42:51,172 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:42:53,984 INFO [make_CHS_currents_file] running in process 1363913
2026-03-14 09:42:53,985 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:42:53,986 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-14 09:42:53,987 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:43:42,457 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-14 for forecast
2026-03-14 09:43:42,461 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-14 09:43:42,630 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:43:44,138 INFO [update_forecast_datasets] running in process 1363951
2026-03-14 09:43:44,139 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:43:44,140 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-14 09:43:44,141 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:43:44,143 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/14mar26 run
2026-03-14 09:43:44,536 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/14mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-14 09:43:44,538 INFO [update_forecast_datasets] nemo 2026-03-14 forecast rolling forecast datasets updated
2026-03-14 09:43:44,541 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-14 09:43:44,671 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-14 09:43:44,676 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:43:44,681 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:43:45,908 INFO [ping_erddap] running in process 1363956
2026-03-14 09:43:45,909 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:43:45,910 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-14 09:43:45,911 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:43:45,923 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-14 09:43:45,925 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 09:43:47,994 INFO [make_surface_current_tiles] running in process 1363959
2026-03-14 09:43:47,996 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:43:47,997 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-14 09:43:47,998 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:43:48,001 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-14 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/14mar26
2026-03-14 09:43:49,732 INFO [make_plots] running in process 1363958
2026-03-14 09:43:49,733 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:43:49,734 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-14 09:43:49,736 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:43:50,127 INFO [make_plots] preparing render list for 2026-03-14 NEMO forecast publish figures
2026-03-14 09:43:50,412 INFO [make_plots] added Website_thumbnail figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,412 INFO [make_plots] added Threshold_website figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,413 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,414 INFO [make_plots] added Boundary Bay figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,415 INFO [make_plots] added Campbell River figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,416 INFO [make_plots] added Cherry Point figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,417 INFO [make_plots] added Friday Harbor figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,417 INFO [make_plots] added Halfmoon Bay figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,418 INFO [make_plots] added Nanaimo figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,419 INFO [make_plots] added Neah Bay figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,420 INFO [make_plots] added New Westminster figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,421 INFO [make_plots] added Patricia Bay figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,421 INFO [make_plots] added Point Atkinson figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,422 INFO [make_plots] added Port Renfrew figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,423 INFO [make_plots] added Sand Heads figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,424 INFO [make_plots] added Sandy Cove figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,425 INFO [make_plots] added Squamish figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,426 INFO [make_plots] added Victoria figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,427 INFO [make_plots] added Woodwards Landing figure to 2026-03-14 NEMO forecast publish render list
2026-03-14 09:43:50,429 INFO [make_plots] starting to render nemo forecast publish 14mar26 figures
2026-03-14 09:44:43,909 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-14 forecast run
2026-03-14 09:44:43,918 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-14 09:44:44,238 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 09:44:46,530 INFO [run_ww3] running in process 20344
2026-03-14 09:44:46,531 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:44:46,532 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-14 09:44:46,533 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:44:46,536 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-14T094446.527501-0700
2026-03-14 09:44:46,537 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-14T094446.527501-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-14 09:44:46,551 INFO [run_ww3] nowcast WaveWatch3 run for 2026-03-14 on arbutus.cloud-nowcast started
2026-03-14 09:44:46,554 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-14 09:44:46,740 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-14 09:44:48,820 INFO [watch_ww3] running in process 20431
2026-03-14 09:44:48,821 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:44:48,822 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-14 09:44:48,822 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:44:48,854 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-14 09:47:25,891 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19096733 = 2026-03-14 01:54:40 UTC, 8.0% complete
2026-03-14 09:49:48,855 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete
2026-03-14 09:51:46,946 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-14 09:51:46,947 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-14 09:52:09,693 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:52:26,025 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19096924 = 2026-03-14 04:02:00 UTC, 16.8% complete
2026-03-14 09:53:15,568 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-14 09:53:15,569 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-14 09:53:37,479 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:54:15,446 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:54:19,569 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/14mar26
2026-03-14 09:54:32,026 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:54:48,648 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:54:48,858 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete
2026-03-14 09:55:04,701 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:55:20,422 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:55:36,485 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:55:53,267 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-14 09:55:53,267 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-14 09:56:08,812 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:56:25,861 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-09 00:00:00Z to 2026-03-16 11:55:00Z
2026-03-14 09:56:42,242 INFO [make_plots] finished rendering nemo forecast publish 14mar26 figures
2026-03-14 09:56:42,244 INFO [make_plots] nemo publish plots for 2026-03-14 forecast completed
2026-03-14 09:56:42,253 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-14 09:56:42,417 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 09:56:47,015 INFO [make_feeds] running in process 1368587
2026-03-14 09:56:47,016 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 09:56:47,017 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-14 09:56:47,018 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-14 09:57:01,412 INFO [make_feeds] ATOM feeds for 2026-03-14 forecast run completed
2026-03-14 09:57:01,416 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-14 09:57:25,896 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19097117 = 2026-03-14 06:10:40 UTC, 25.8% complete
2026-03-14 09:59:07,159 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/14mar26
2026-03-14 09:59:07,184 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-14 forecast into /results/nowcast-sys/figures/surface_currents/forecast/14mar26
2026-03-14 09:59:07,185 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-14 forecast completed
2026-03-14 09:59:07,644 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-14 09:59:48,863 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 291, 67.4% complete
2026-03-14 10:02:25,901 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19097309 = 2026-03-14 08:18:40 UTC, 34.6% complete
2026-03-14 10:04:48,867 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 396, 91.7% complete
2026-03-14 10:07:25,904 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19097502 = 2026-03-14 10:27:20 UTC, 43.6% complete
2026-03-14 10:09:48,875 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-14 10:09:48,989 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-14 10:09:49,392 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:09:49,399 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-14'], host='arbutus.cloud-nowcast')
2026-03-14 10:09:50,647 INFO [download_wwatch3_results] running in process 1368774
2026-03-14 10:09:50,648 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:09:50,649 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-14 10:09:50,651 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:09:52,706 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-14 10:09:53,016 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-14 10:09:55,059 INFO [watch_ww3] running in process 20938
2026-03-14 10:09:55,060 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:09:55,062 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-14 10:09:55,063 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:09:55,097 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-14 10:10:22,849 INFO [download_wwatch3_results] nowcast 2026-03-14 results files from arbutus.cloud-nowcast downloaded
2026-03-14 10:10:22,852 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-14 10:12:25,907 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19097694 = 2026-03-14 12:35:20 UTC, 52.5% complete
2026-03-14 10:14:55,099 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2026-03-14 10:17:25,910 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19097887 = 2026-03-14 14:44:00 UTC, 61.4% complete
2026-03-14 10:19:55,102 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 201, 31.0% complete
2026-03-14 10:22:25,915 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19098079 = 2026-03-14 16:52:00 UTC, 70.3% complete
2026-03-14 10:24:55,104 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 300, 46.3% complete
2026-03-14 10:27:25,915 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19098270 = 2026-03-14 18:59:20 UTC, 79.2% complete
2026-03-14 10:29:55,106 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 399, 61.6% complete
2026-03-14 10:32:25,917 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19098461 = 2026-03-14 21:06:40 UTC, 88.0% complete
2026-03-14 10:34:55,108 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete
2026-03-14 10:37:25,919 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19098653 = 2026-03-14 23:14:40 UTC, 96.9% complete
2026-03-14 10:39:55,105 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 603, 93.1% complete
2026-03-14 10:42:25,912 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19098720 = 2026-03-14 23:59:20 UTC, 100.0% complete
2026-03-14 10:44:55,099 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-14 10:44:55,103 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-14 10:44:55,566 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:44:56,830 INFO [download_wwatch3_results] running in process 1369160
2026-03-14 10:44:56,832 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:44:56,833 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-14 10:44:56,834 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:45:36,388 INFO [download_wwatch3_results] forecast 2026-03-14 results files from arbutus.cloud-nowcast downloaded
2026-03-14 10:45:36,392 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-14 10:45:36,695 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:45:37,817 INFO [update_forecast_datasets] running in process 1369183
2026-03-14 10:45:37,818 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:45:37,819 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-14 10:45:37,820 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:45:37,821 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/14mar26 run
2026-03-14 10:45:37,825 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/14mar26 run
2026-03-14 10:45:37,825 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/14mar26 run
2026-03-14 10:45:37,941 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/14mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-14 10:45:37,942 INFO [update_forecast_datasets] wwatch3 2026-03-14 forecast rolling forecast datasets updated
2026-03-14 10:45:37,945 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-14 10:45:38,203 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-14 10:45:39,183 INFO [ping_erddap] running in process 1369188
2026-03-14 10:45:39,184 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:45:39,186 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-14 10:45:39,187 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:45:39,189 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-14 10:45:39,190 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 10:45:39,442 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:45:42,814 INFO [make_plots] running in process 1369193
2026-03-14 10:45:42,815 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:45:42,817 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-14 10:45:42,818 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:45:42,819 INFO [make_plots] preparing render list for 2026-03-14 WaveWatch3 forecast publish figures
2026-03-14 10:45:42,820 INFO [make_plots] added Halibut Bank figure to 2026-03-14 WaveWatch3 forecast publish render list
2026-03-14 10:45:42,821 INFO [make_plots] added Sentry Shoal figure to 2026-03-14 WaveWatch3 forecast publish render list
2026-03-14 10:45:42,821 INFO [make_plots] starting to render wwatch3 forecast publish 14mar26 figures
2026-03-14 10:45:43,317 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-14 10:45:49,188 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-14 10:45:54,687 INFO [make_plots] finished rendering wwatch3 forecast publish 14mar26 figures
2026-03-14 10:45:54,687 INFO [make_plots] wwatch3 publish plots for 2026-03-14 forecast completed
2026-03-14 10:45:54,691 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-14 10:47:25,920 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-03-14 10:47:25,928 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-14 10:47:26,395 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:47:27,728 INFO [download_results] running in process 1369227
2026-03-14 10:47:27,730 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:47:27,731 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-14 10:47:27,732 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:47:27,733 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/14mar26 to /results2/SalishSea/nowcast-green.202111
2026-03-14 10:49:30,292 INFO [download_results] nowcast-green 2026-03-14 results files from arbutus.cloud-nowcast downloaded
2026-03-14 10:49:30,299 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-14 10:49:30,615 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:49:30,621 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-03-14 10:49:30,626 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:49:30,631 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:49:30,636 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-03-14'], host='localhost')
2026-03-14 10:49:31,922 INFO [ping_erddap] running in process 1369253
2026-03-14 10:49:31,924 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:49:31,924 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-14 10:49:31,925 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:49:31,931 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-03-14 10:49:31,933 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-14 10:49:33,257 INFO [make_averaged_dataset] running in process 1369256
2026-03-14 10:49:33,259 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:49:33,260 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-03-14 10:49:33,261 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:49:33,262 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Mar-2026 chemistry
2026-03-14 10:49:33,263 INFO [make_averaged_dataset] running in process 1369255
2026-03-14 10:49:33,263 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:49:33,264 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-03-14 10:49:33,265 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:49:33,266 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Mar-2026 biology
2026-03-14 10:49:33,284 INFO [make_averaged_dataset] running in process 1369254
2026-03-14 10:49:33,285 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:49:33,286 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-03-14 10:49:33,287 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:49:33,288 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Mar-2026 physics
2026-03-14 10:49:34,067 INFO [make_plots] running in process 1369251
2026-03-14 10:49:34,069 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 10:49:34,070 INFO [make_plots] publishing log messages to tcp://*:5566
2026-03-14 10:49:34,071 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-14 10:49:34,463 INFO [make_plots] preparing render list for 2026-03-14 NEMO nowcast-green research figures
2026-03-14 10:49:36,200 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:36,946 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:37,675 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:38,448 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:39,205 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:39,955 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:40,725 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:41,584 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:42,451 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:43,280 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:44,122 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:44,926 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:46,014 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:46,014 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:46,015 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:46,016 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-03-14 NEMO nowcast-green research render list
2026-03-14 10:49:46,017 INFO [make_plots] starting to render nemo nowcast-green research 14mar26 figures
2026-03-14 10:50:07,089 INFO [make_averaged_dataset] day-averaged dataset for 14-Mar-2026 physics created
2026-03-14 10:50:07,092 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-14 10:50:30,774 INFO [make_averaged_dataset] day-averaged dataset for 14-Mar-2026 chemistry created
2026-03-14 10:50:30,777 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-14 10:50:40,454 INFO [make_averaged_dataset] day-averaged dataset for 14-Mar-2026 biology created
2026-03-14 10:50:40,457 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-14 11:53:54,874 INFO [make_plots] finished rendering nemo nowcast-green research 14mar26 figures
2026-03-14 11:53:55,066 INFO [make_plots] nemo research plots for 2026-03-14 nowcast-green completed
2026-03-14 11:53:55,129 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-14 14:43:50,317 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260314/18/
2026-03-14 14:43:50,317 INFO [collect_weather] 2026-03-14 2.5km weather forecast 18 collection complete
2026-03-14 14:43:50,321 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-14 14:43:50,805 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-03-14 14:43:50,812 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-03-14 14:43:50,816 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-03-14 14:43:50,821 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-03-15'], host='localhost')
2026-03-14 14:43:52,494 INFO [collect_weather] running in process 1372028
2026-03-14 14:43:52,495 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 14:43:52,495 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-14 14:43:52,496 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-14 14:43:52,501 INFO [download_weather] running in process 1372027
2026-03-14 14:43:52,502 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 14:43:52,502 INFO [download_weather] publishing log messages to tcp://*:5563
2026-03-14 14:43:52,503 INFO [download_weather] running in process 1372025
2026-03-14 14:43:52,504 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-14 14:43:52,504 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260314
2026-03-14 14:43:52,505 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 14:43:52,505 INFO [download_weather] publishing log messages to tcp://*:5564
2026-03-14 14:43:52,506 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-14 14:43:52,507 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260314
2026-03-14 14:43:52,551 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-03-14 14:43:53,665 INFO [crop_gribs] running in process 1372029
2026-03-14 14:43:53,666 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-14 14:43:53,667 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-03-14 14:43:53,668 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-14 14:43:53,668 INFO [crop_gribs] cropping 2026-03-15 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-03-14 14:43:53,684 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260315/00/
2026-03-14 14:47:33,425 INFO [download_weather] 2026-03-14 1km weather forecast 00 downloads complete
2026-03-14 14:47:33,428 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-14 14:47:33,964 INFO [download_weather] 2026-03-14 1km weather forecast 12 downloads complete
2026-03-14 14:47:33,968 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-14 16:40:57,163 INFO [crop_gribs] crop_gribs 18 has watched for 8h; retrying remaining 1 file(s)
2026-03-14 16:40:57,880 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260314/18/
2026-03-14 16:40:57,881 INFO [crop_gribs] 2026-03-14 18 GRIBs cropping complete
2026-03-14 16:40:57,883 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker