2026-03-10 03:37:42,006 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-03-10 03:37:42,016 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-03-10 03:37:42,021 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-03-10 03:37:42,022 INFO [rotate_logs] log files rotated
2026-03-10 03:37:42,027 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-03-10 03:39:16,740 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 390, 72.2% complete
2026-03-10 03:39:27,279 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/09mar26
2026-03-10 03:39:27,303 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-09 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/09mar26
2026-03-10 03:39:27,304 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-09 forecast2 completed
2026-03-10 03:39:27,598 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-10 03:44:16,742 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 489, 90.6% complete
2026-03-10 03:49:16,743 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-10 03:49:16,746 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-10 03:49:17,134 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 03:49:18,477 INFO [download_wwatch3_results] running in process 1235699
2026-03-10 03:49:18,478 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:49:18,479 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-03-10 03:49:18,480 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:49:55,028 INFO [download_wwatch3_results] forecast2 2026-03-10 results files from arbutus.cloud-nowcast downloaded
2026-03-10 03:49:55,032 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-10 03:49:55,182 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 03:49:56,519 INFO [update_forecast_datasets] running in process 1235710
2026-03-10 03:49:56,521 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:49:56,522 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-10 03:49:56,523 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:49:56,524 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/10mar26 run
2026-03-10 03:49:56,529 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/10mar26 run
2026-03-10 03:49:56,530 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/10mar26 run
2026-03-10 03:50:23,357 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/10mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-10 03:50:23,358 INFO [update_forecast_datasets] wwatch3 2026-03-10 forecast2 rolling forecast datasets updated
2026-03-10 03:50:23,362 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-10 03:50:23,512 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-10 03:50:24,773 INFO [ping_erddap] running in process 1235722
2026-03-10 03:50:24,774 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:50:24,775 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-10 03:50:24,777 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:50:24,778 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-10 03:50:24,781 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 03:50:24,925 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 03:50:28,740 INFO [make_plots] running in process 1235727
2026-03-10 03:50:28,742 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 03:50:28,743 INFO [make_plots] publishing log messages to tcp://*:5563
2026-03-10 03:50:28,745 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-10 03:50:28,746 INFO [make_plots] preparing render list for 2026-03-10 WaveWatch3 forecast2 publish figures
2026-03-10 03:50:28,747 INFO [make_plots] added Halibut Bank figure to 2026-03-10 WaveWatch3 forecast2 publish render list
2026-03-10 03:50:28,748 INFO [make_plots] added Sentry Shoal figure to 2026-03-10 WaveWatch3 forecast2 publish render list
2026-03-10 03:50:28,749 INFO [make_plots] starting to render wwatch3 forecast2 publish 10mar26 figures
2026-03-10 03:50:29,228 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-10 03:50:33,966 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-10 03:50:38,915 INFO [make_plots] finished rendering wwatch3 forecast2 publish 10mar26 figures
2026-03-10 03:50:38,915 INFO [make_plots] wwatch3 publish plots for 2026-03-10 forecast2 completed
2026-03-10 03:50:38,919 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-10 08:43:49,937 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260310/12/
2026-03-10 08:43:49,938 INFO [collect_weather] 2026-03-10 2.5km weather forecast 12 collection complete
2026-03-10 08:43:49,942 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-10 08:43:50,267 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 08:43:50,329 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 08:43:50,336 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 08:43:50,341 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-03-09'], host='localhost')
2026-03-10 08:43:50,345 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-03-10 08:43:50,352 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-03-10 08:43:50,355 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-03-10 08:43:50,364 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-03-10 08:43:50,371 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-03-10 08:43:50,980 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260310/12/
2026-03-10 08:43:50,981 INFO [crop_gribs] 2026-03-10 12 GRIBs cropping complete
2026-03-10 08:43:50,984 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-10 08:43:51,380 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-03-10 08:43:52,649 INFO [download_live_ocean] running in process 1239836
2026-03-10 08:43:52,651 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:52,651 INFO [download_live_ocean] publishing log messages to tcp://*:5562
2026-03-10 08:43:52,655 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:52,656 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-03-10
2026-03-10 08:43:52,657 INFO [collect_weather] running in process 1239838
2026-03-10 08:43:52,657 INFO [collect_NeahBay_ssh] running in process 1239835
2026-03-10 08:43:52,658 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:52,659 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:52,659 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-03-10 08:43:52,660 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2026-03-10 08:43:52,661 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:52,663 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:52,664 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-03-10
2026-03-10 08:43:52,676 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-03-10 08:43:53,193 INFO [download_live_ocean] downloaded 10210149 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.03.10/ubc.nc to /results/forcing/LiveOcean/downloaded/20260310/low_passed_UBC.nc
2026-03-10 08:43:53,194 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-03-10 08:43:53,368 INFO [collect_river_data] running in process 1239833
2026-03-10 08:43:53,372 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:53,373 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-10 08:43:53,374 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:53,374 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-03-09
2026-03-10 08:43:53,444 INFO [collect_river_data] running in process 1239831
2026-03-10 08:43:53,445 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:53,446 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-10 08:43:53,447 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:53,449 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-03-09
2026-03-10 08:43:53,468 INFO [collect_river_data] running in process 1239832
2026-03-10 08:43:53,469 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:53,470 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-10 08:43:53,471 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:53,472 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-03-09
2026-03-10 08:43:53,474 INFO [collect_river_data] running in process 1239829
2026-03-10 08:43:53,475 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:53,476 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-10 08:43:53,480 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:53,481 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-03-09
2026-03-10 08:43:53,553 INFO [make_turbidity_file] running in process 1239834
2026-03-10 08:43:53,554 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:53,555 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-03-10 08:43:53,556 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:53,556 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-03-10
2026-03-10 08:43:53,806 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-03-09&endDT=2026-03-09 "HTTP/1.1 200 "
2026-03-10 08:43:53,813 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-03-09&endDT=2026-03-09 "HTTP/1.1 200 "
2026-03-10 08:43:53,859 INFO [grib_to_netcdf] running in process 1239913
2026-03-10 08:43:53,860 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:53,861 INFO [grib_to_netcdf] publishing log messages to tcp://*:5570
2026-03-10 08:43:53,862 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:53,863 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-10 nowcast and forecast runs
2026-03-10 08:43:53,964 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-03-09&endDT=2026-03-09 "HTTP/1.1 200 "
2026-03-10 08:43:53,974 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-03-09&endDT=2026-03-09 "HTTP/1.1 200 "
2026-03-10 08:43:53,987 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-03-10 08:43:53,988 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-03-09 completed
2026-03-10 08:43:53,993 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-03-10 08:43:53,994 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 08:43:53,995 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-03-09 completed
2026-03-10 08:43:54,012 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-03-10 08:43:54,013 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-03-09 completed
2026-03-10 08:43:54,032 INFO [crop_gribs] running in process 1239839
2026-03-10 08:43:54,034 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:54,035 INFO [crop_gribs] publishing log messages to tcp://*:5571
2026-03-10 08:43:54,036 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:54,037 INFO [crop_gribs] cropping 2026-03-10 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-03-10 08:43:54,054 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260310/18/
2026-03-10 08:43:54,057 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-03-09 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-03-10 08:43:54,058 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-03-09 completed
2026-03-10 08:43:54,135 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 08:43:54,212 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260310/low_passed_UBC.nc
2026-03-10 08:43:54,213 INFO [download_live_ocean] 2026-03-10 Live Ocean file for Salish Sea western boundary downloaded
2026-03-10 08:43:54,308 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 08:43:54,484 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-10 08:43:54,653 INFO [collect_NeahBay_ssh] 2026-03-10T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-03-10 08:43:54,655 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-03-10 08:43:54,985 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-03-10'], host='localhost')
2026-03-10 08:43:54,992 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-10 08:43:55,149 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 08:43:56,635 INFO [make_turbidity_file] 2026-03-10 Fraser River turbidity file creation complete
2026-03-10 08:43:56,636 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-03-10 08:43:58,399 INFO [make_live_ocean_files] running in process 1239983
2026-03-10 08:43:58,402 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:58,403 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2026-03-10 08:43:58,404 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:58,404 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-03-10 Live Ocean run
2026-03-10 08:43:58,830 INFO [make_ssh_files] running in process 1239985
2026-03-10 08:43:58,831 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:43:58,832 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2026-03-10 08:43:58,833 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:43:58,834 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-03-10 NOAA Neah Bay 06Z observation and forecast values
2026-03-10 08:44:00,804 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d06.nc
2026-03-10 08:44:00,979 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d07.nc
2026-03-10 08:44:01,171 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d08.nc
2026-03-10 08:44:01,348 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d09.nc
2026-03-10 08:44:01,590 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d10.nc
2026-03-10 08:44:01,821 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d11.nc
2026-03-10 08:44:02,008 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d12.nc
2026-03-10 08:44:02,213 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d13.nc
2026-03-10 08:44:02,357 INFO [make_ssh_files] sea surface height boundary file for 2026-03-10 nowcast run created
2026-03-10 08:44:02,361 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-10 08:44:02,511 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-10 08:44:02,515 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-10 08:44:03,947 INFO [make_201702_runoff_file] running in process 1240063
2026-03-10 08:44:03,948 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:03,950 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2026-03-10 08:44:03,951 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:03,991 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-10 08:44:04,124 INFO [make_201702_runoff_file] 2026-03-10 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-10 08:44:04,126 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-10 08:44:04,677 INFO [make_runoff_file] running in process 1240061
2026-03-10 08:44:04,678 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:04,679 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2026-03-10 08:44:04,680 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:04,683 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-09
2026-03-10 08:44:07,778 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m03d10.nc
2026-03-10 08:44:07,887 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-09: /results/forcing/rivers/R202108Dailies_y2026m03d09.nc
2026-03-10 08:44:07,888 INFO [make_runoff_file] 2026-03-09 runoff file creation completed
2026-03-10 08:44:07,890 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-10 08:44:17,146 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d11.nc
2026-03-10 08:44:22,510 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d12.nc
2026-03-10 08:44:22,525 INFO [grib_to_netcdf] 2026-03-10 NEMO-atmos forcing file for nowcast+ created
2026-03-10 08:44:22,527 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-10 08:44:40,239 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d10.nc
2026-03-10 08:44:40,240 INFO [make_live_ocean_files] 2026-03-10 Live Ocean western boundary conditions files created
2026-03-10 08:44:40,242 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-03-10 08:44:40,371 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-03-10 08:44:40,377 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-03-10 08:44:40,383 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2026-03-10 08:44:40,387 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-03-10 08:44:40,391 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-03-10 08:44:41,676 INFO [ping_erddap] running in process 1240137
2026-03-10 08:44:41,677 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:41,678 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-10 08:44:41,680 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:41,681 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-03-10 08:44:41,683 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 08:44:41,890 INFO [upload_forcing] running in process 1240142
2026-03-10 08:44:41,891 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:41,892 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-10 08:44:41,893 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:41,896 INFO [upload_forcing] running in process 1240141
2026-03-10 08:44:41,897 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:41,898 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-10 08:44:41,899 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:41,900 INFO [upload_forcing] running in process 1240140
2026-03-10 08:44:41,901 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:41,901 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-10 08:44:41,902 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:41,903 INFO [upload_forcing] running in process 1240139
2026-03-10 08:44:41,904 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:41,905 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-10 08:44:41,906 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:43,781 INFO [upload_forcing] nowcast+ 2026-03-10 forcing files upload to optimum-hindcast completed
2026-03-10 08:44:43,784 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 08:44:45,807 INFO [upload_forcing] nowcast+ 2026-03-10 forcing files upload to arbutus.cloud-nowcast completed
2026-03-10 08:44:45,811 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 08:44:45,984 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 08:44:47,423 INFO [make_forcing_links] running in process 1240175
2026-03-10 08:44:47,424 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:47,425 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-03-10 08:44:47,427 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:48,727 INFO [make_forcing_links] nowcast+ 2026-03-10 forcing file links on arbutus.cloud-nowcast created
2026-03-10 08:44:48,731 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-10 08:44:48,917 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 08:44:49,525 INFO [upload_forcing] nowcast+ 2026-03-10 forcing files upload to orcinus-nowcast-agrif completed
2026-03-10 08:44:49,529 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 08:44:51,017 INFO [run_NEMO] running in process 12699
2026-03-10 08:44:51,018 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:51,019 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-10 08:44:51,031 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:51,737 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-10 08:44:52,047 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/10mar26nowcast_2026-03-10T084451.059603-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-10 08:44:52,061 INFO [run_NEMO] nowcast NEMO run for 2026-03-10 on arbutus.cloud-nowcast started
2026-03-10 08:44:52,065 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-10 08:44:52,220 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-10 08:44:53,653 INFO [upload_forcing] nowcast+ 2026-03-10 forcing files upload to robot.nibi completed
2026-03-10 08:44:53,658 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 08:44:54,274 INFO [watch_NEMO] running in process 12974
2026-03-10 08:44:54,275 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 08:44:54,276 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-10 08:44:54,278 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-10 08:44:54,343 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-10 08:49:54,348 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19088384 = 2026-03-10 05:08:40 UTC, 21.4% complete
2026-03-10 08:54:54,350 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19088938 = 2026-03-10 11:18:00 UTC, 47.1% complete
2026-03-10 08:59:54,352 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19089501 = 2026-03-10 17:33:20 UTC, 73.2% complete
2026-03-10 09:04:54,354 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19090031 = 2026-03-10 23:26:40 UTC, 97.7% complete
2026-03-10 09:09:54,359 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-03-10 09:09:54,362 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-10 09:09:54,690 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:09:54,696 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:09:55,884 INFO [download_results] running in process 1240444
2026-03-10 09:09:55,885 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:09:55,886 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-10 09:09:55,887 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:09:55,888 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/10mar26 to /results/SalishSea/nowcast-blue.202111
2026-03-10 09:09:56,031 INFO [make_forcing_links] running in process 1240443
2026-03-10 09:09:56,032 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:09:56,034 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-10 09:09:56,035 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:09:57,864 INFO [make_forcing_links] ssh 2026-03-10 forcing file links on arbutus.cloud-nowcast created
2026-03-10 09:09:57,868 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-10 09:09:58,025 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 09:10:00,115 INFO [run_NEMO] running in process 13588
2026-03-10 09:10:00,116 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:10:00,117 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-10 09:10:00,119 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:10:00,893 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-10 09:10:01,217 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/10mar26forecast_2026-03-10T091000.168382-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-10 09:10:01,222 INFO [run_NEMO] forecast NEMO run for 2026-03-10 on arbutus.cloud-nowcast started
2026-03-10 09:10:01,226 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-10 09:10:01,383 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-10 09:10:03,477 INFO [watch_NEMO] running in process 13861
2026-03-10 09:10:03,480 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:10:03,481 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-10 09:10:03,482 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:10:03,557 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-10 09:10:31,040 INFO [download_results] nowcast 2026-03-10 results files from arbutus.cloud-nowcast downloaded
2026-03-10 09:10:31,045 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-10 09:10:31,203 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:10:31,207 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-03-09'], host='localhost')
2026-03-10 09:10:31,213 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:10:33,221 INFO [make_CHS_currents_file] running in process 1240469
2026-03-10 09:10:33,223 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:10:33,224 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-10 09:10:33,224 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:10:34,726 INFO [make_plots] running in process 1240468
2026-03-10 09:10:34,728 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:10:34,729 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-10 09:10:34,730 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:10:34,743 INFO [make_plots] running in process 1240466
2026-03-10 09:10:34,746 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:10:34,747 INFO [make_plots] publishing log messages to tcp://*:5565
2026-03-10 09:10:34,748 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:10:35,246 INFO [make_plots] preparing render list for 2026-03-09 NEMO nowcast-blue comparison figures
2026-03-10 09:10:35,248 INFO [make_plots] preparing render list for 2026-03-10 NEMO nowcast-blue research figures
2026-03-10 09:10:35,318 INFO [make_plots] added SH_wind figure to 2026-03-09 NEMO nowcast-blue comparison render list
2026-03-10 09:10:35,319 INFO [make_plots] added Compare_VENUS_East figure to 2026-03-09 NEMO nowcast-blue comparison render list
2026-03-10 09:10:35,319 INFO [make_plots] added Compare_VENUS_Central figure to 2026-03-09 NEMO nowcast-blue comparison render list
2026-03-10 09:10:35,320 INFO [make_plots] starting to render nemo nowcast comparison 09mar26 figures
2026-03-10 09:10:36,597 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-blue research render list
2026-03-10 09:10:37,685 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-blue research render list
2026-03-10 09:10:37,686 INFO [make_plots] added Currents_sections_and_surface figure to 2026-03-10 NEMO nowcast-blue research render list
2026-03-10 09:10:37,687 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-03-10 NEMO nowcast-blue research render list
2026-03-10 09:10:37,688 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-03-10 NEMO nowcast-blue research render list
2026-03-10 09:10:37,689 INFO [make_plots] starting to render nemo nowcast research 10mar26 figures
2026-03-10 09:11:07,845 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-10 for nowcast
2026-03-10 09:11:07,848 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-10 09:11:41,415 INFO [make_plots] finished rendering nemo nowcast research 10mar26 figures
2026-03-10 09:11:41,417 INFO [make_plots] nemo research plots for 2026-03-10 nowcast completed
2026-03-10 09:11:41,433 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-10 09:12:03,861 INFO [make_plots] finished rendering nemo nowcast comparison 09mar26 figures
2026-03-10 09:12:03,863 INFO [make_plots] nemo comparison plots for 2026-03-09 nowcast completed
2026-03-10 09:12:03,868 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-10 09:15:03,557 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19090545 = 2026-03-11 05:09:20 UTC, 14.3% complete
2026-03-10 09:20:03,560 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19091085 = 2026-03-11 11:09:20 UTC, 31.0% complete
2026-03-10 09:25:03,561 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19091606 = 2026-03-11 16:56:40 UTC, 47.1% complete
2026-03-10 09:30:03,565 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19092125 = 2026-03-11 22:42:40 UTC, 63.1% complete
2026-03-10 09:35:03,563 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19092684 = 2026-03-12 04:55:20 UTC, 80.4% complete
2026-03-10 09:40:03,561 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19093217 = 2026-03-12 10:50:40 UTC, 96.8% complete
2026-03-10 09:45:03,564 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-03-10 09:45:03,568 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-10 09:45:03,895 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 09:45:03,909 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 09:45:03,914 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:45:03,920 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:45:03,926 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:45:03,929 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:45:03,940 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:45:05,513 INFO [download_results] running in process 1240898
2026-03-10 09:45:05,514 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:05,515 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-10 09:45:05,516 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:05,517 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/10mar26 to /results/SalishSea/forecast.202111
2026-03-10 09:45:05,554 INFO [upload_forcing] running in process 1240894
2026-03-10 09:45:05,555 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:05,557 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-10 09:45:05,558 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:05,564 INFO [upload_forcing] running in process 1240896
2026-03-10 09:45:05,566 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:05,568 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-03-10 09:45:05,570 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:05,575 INFO [upload_forcing] running in process 1240897
2026-03-10 09:45:05,577 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:05,578 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-03-10 09:45:05,579 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:05,582 INFO [upload_forcing] running in process 1240895
2026-03-10 09:45:05,583 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:05,584 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-10 09:45:05,585 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:06,276 INFO [upload_forcing] turbidity 2026-03-10 forcing files upload to optimum-hindcast completed
2026-03-10 09:45:06,280 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 09:45:06,585 INFO [upload_forcing] turbidity 2026-03-10 forcing files upload to arbutus.cloud-nowcast completed
2026-03-10 09:45:06,589 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 09:45:06,752 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:45:06,947 INFO [make_ww3_wind_file] running in process 15252
2026-03-10 09:45:06,949 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:06,950 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-10 09:45:07,010 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:07,011 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-10 forecast run
2026-03-10 09:45:07,033 INFO [make_ww3_current_file] running in process 15269
2026-03-10 09:45:07,034 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:07,036 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-10 09:45:07,038 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:07,041 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-10 forecast run
2026-03-10 09:45:08,255 INFO [make_forcing_links] running in process 1240929
2026-03-10 09:45:08,256 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:08,258 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-03-10 09:45:08,259 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:09,433 INFO [make_forcing_links] nowcast-green 2026-03-10 forcing file links on arbutus.cloud-nowcast created
2026-03-10 09:45:09,438 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-10 09:45:09,599 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 09:45:10,575 INFO [upload_forcing] turbidity 2026-03-10 forcing files upload to robot.nibi completed
2026-03-10 09:45:10,579 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 09:45:11,727 INFO [run_NEMO] running in process 15603
2026-03-10 09:45:11,729 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:11,730 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-10 09:45:11,731 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:11,773 INFO [upload_forcing] turbidity 2026-03-10 forcing files upload to orcinus-nowcast-agrif completed
2026-03-10 09:45:11,778 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-10 09:45:14,243 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-10 09:45:15,327 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/10mar26nowcast-green_2026-03-10T094511.858959-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-10 09:45:15,392 INFO [run_NEMO] nowcast-green NEMO run for 2026-03-10 on arbutus.cloud-nowcast started
2026-03-10 09:45:15,397 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-10 09:45:15,564 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-03-10 09:45:19,352 INFO [watch_NEMO] running in process 16178
2026-03-10 09:45:19,353 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:45:19,355 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-10 09:45:19,356 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:45:19,370 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-10 forecast run
2026-03-10 09:45:19,374 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-10 09:45:19,423 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-10 09:46:11,475 INFO [download_results] forecast 2026-03-10 results files from arbutus.cloud-nowcast downloaded
2026-03-10 09:46:11,479 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-10 09:46:11,651 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:46:14,828 INFO [make_CHS_currents_file] running in process 1240953
2026-03-10 09:46:14,829 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:46:14,830 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-03-10 09:46:14,832 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:47:03,102 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-10 for forecast
2026-03-10 09:47:03,105 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-10 09:47:03,270 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:47:04,455 INFO [update_forecast_datasets] running in process 1240990
2026-03-10 09:47:04,456 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:47:04,457 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-03-10 09:47:04,458 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:47:04,459 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/10mar26 run
2026-03-10 09:47:04,712 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/10mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-10 09:47:04,713 INFO [update_forecast_datasets] nemo 2026-03-10 forecast rolling forecast datasets updated
2026-03-10 09:47:04,716 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-10 09:47:04,884 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-10 09:47:04,896 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:47:04,907 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 09:47:06,250 INFO [ping_erddap] running in process 1240997
2026-03-10 09:47:06,251 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:47:06,252 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-10 09:47:06,254 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:47:06,267 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-10 09:47:06,269 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 09:47:08,049 INFO [make_surface_current_tiles] running in process 1241000
2026-03-10 09:47:08,051 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:47:08,052 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-03-10 09:47:08,053 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:47:08,055 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-10 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/10mar26
2026-03-10 09:47:09,367 INFO [make_plots] running in process 1240999
2026-03-10 09:47:09,372 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:47:09,386 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-10 09:47:09,387 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:47:09,899 INFO [make_plots] preparing render list for 2026-03-10 NEMO forecast publish figures
2026-03-10 09:47:10,297 INFO [make_plots] added Website_thumbnail figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,300 INFO [make_plots] added Threshold_website figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,304 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,305 INFO [make_plots] added Boundary Bay figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,306 INFO [make_plots] added Campbell River figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,307 INFO [make_plots] added Cherry Point figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,308 INFO [make_plots] added Friday Harbor figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,309 INFO [make_plots] added Halfmoon Bay figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,310 INFO [make_plots] added Nanaimo figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,312 INFO [make_plots] added Neah Bay figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,313 INFO [make_plots] added New Westminster figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,315 INFO [make_plots] added Patricia Bay figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,316 INFO [make_plots] added Point Atkinson figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,317 INFO [make_plots] added Port Renfrew figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,318 INFO [make_plots] added Sand Heads figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,319 INFO [make_plots] added Sandy Cove figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,320 INFO [make_plots] added Squamish figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,321 INFO [make_plots] added Victoria figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,321 INFO [make_plots] added Woodwards Landing figure to 2026-03-10 NEMO forecast publish render list
2026-03-10 09:47:10,322 INFO [make_plots] starting to render nemo forecast publish 10mar26 figures
2026-03-10 09:47:42,276 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-10 forecast run
2026-03-10 09:47:42,281 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-10 09:47:42,596 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 09:47:44,839 INFO [run_ww3] running in process 16301
2026-03-10 09:47:44,840 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:47:44,841 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-10 09:47:44,842 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:47:44,847 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-10T094744.828605-0700
2026-03-10 09:47:44,849 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-03-10T094744.828605-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-10 09:47:44,859 INFO [run_ww3] nowcast WaveWatch3 run for 2026-03-10 on arbutus.cloud-nowcast started
2026-03-10 09:47:44,863 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-10 09:47:45,175 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-03-10 09:47:47,188 INFO [watch_ww3] running in process 16388
2026-03-10 09:47:47,189 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 09:47:47,190 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-10 09:47:47,191 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-10 09:47:47,225 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-10 09:50:19,424 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19088094 = 2026-03-10 01:55:20 UTC, 8.0% complete
2026-03-10 09:52:47,224 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2026-03-10 09:55:15,296 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-10 09:55:15,297 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-10 09:55:19,424 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19088287 = 2026-03-10 04:04:00 UTC, 17.0% complete
2026-03-10 09:55:35,658 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:56:38,168 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-10 09:56:38,169 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-10 09:56:59,609 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:57:25,796 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/10mar26
2026-03-10 09:57:34,353 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:57:47,223 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 198, 45.8% complete
2026-03-10 09:57:51,285 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:58:08,263 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:58:24,451 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:58:46,042 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:59:01,962 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:59:18,661 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-10 09:59:18,662 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-10 09:59:34,077 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 09:59:51,429 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-05 00:00:00Z to 2026-03-12 11:55:00Z
2026-03-10 10:00:08,193 INFO [make_plots] finished rendering nemo forecast publish 10mar26 figures
2026-03-10 10:00:08,195 INFO [make_plots] nemo publish plots for 2026-03-10 forecast completed
2026-03-10 10:00:08,201 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-10 10:00:08,371 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:00:13,423 INFO [make_feeds] running in process 1245671
2026-03-10 10:00:13,425 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:00:13,426 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-10 10:00:13,427 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:00:19,422 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19088480 = 2026-03-10 06:12:40 UTC, 25.9% complete
2026-03-10 10:00:28,422 INFO [make_feeds] ATOM feeds for 2026-03-10 forecast run completed
2026-03-10 10:00:28,426 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-10 10:02:16,877 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/10mar26
2026-03-10 10:02:16,902 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-03-10 forecast into /results/nowcast-sys/figures/surface_currents/forecast/10mar26
2026-03-10 10:02:16,903 INFO [make_surface_current_tiles] surface current tile figures for 2026-03-10 forecast completed
2026-03-10 10:02:17,207 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-03-10 10:02:47,218 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 297, 68.8% complete
2026-03-10 10:05:19,418 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19088673 = 2026-03-10 08:21:20 UTC, 34.8% complete
2026-03-10 10:07:47,215 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 399, 92.4% complete
2026-03-10 10:10:19,416 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19088866 = 2026-03-10 10:30:00 UTC, 43.8% complete
2026-03-10 10:12:47,215 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-10 10:12:47,218 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-10 10:12:47,707 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:12:47,715 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-10'], host='arbutus.cloud-nowcast')
2026-03-10 10:12:48,956 INFO [download_wwatch3_results] running in process 1245887
2026-03-10 10:12:48,957 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:12:48,959 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-03-10 10:12:48,960 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:12:49,698 INFO [run_ww3] running in process 16747
2026-03-10 10:12:49,699 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:12:49,700 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-10 10:12:49,701 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:12:49,706 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-10T101249.701154-0700
2026-03-10 10:12:49,708 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-03-10T101249.701154-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-10 10:12:49,719 INFO [run_ww3] forecast WaveWatch3 run for 2026-03-10 on arbutus.cloud-nowcast started
2026-03-10 10:12:49,723 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-10 10:12:50,022 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-03-10 10:12:52,034 INFO [watch_ww3] running in process 16902
2026-03-10 10:12:52,036 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:12:52,037 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-10 10:12:52,038 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:12:52,070 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-10 10:13:22,595 INFO [download_wwatch3_results] nowcast 2026-03-10 results files from arbutus.cloud-nowcast downloaded
2026-03-10 10:13:22,598 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-10 10:15:19,415 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19089060 = 2026-03-10 12:39:20 UTC, 52.8% complete
2026-03-10 10:17:52,069 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2026-03-10 10:20:19,415 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19089253 = 2026-03-10 14:48:00 UTC, 61.7% complete
2026-03-10 10:22:52,069 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2026-03-10 10:25:19,416 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19089446 = 2026-03-10 16:56:40 UTC, 70.6% complete
2026-03-10 10:27:52,071 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 291, 44.9% complete
2026-03-10 10:30:19,420 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19089638 = 2026-03-10 19:04:40 UTC, 79.5% complete
2026-03-10 10:32:52,074 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 390, 60.2% complete
2026-03-10 10:35:19,421 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19089831 = 2026-03-10 21:13:20 UTC, 88.5% complete
2026-03-10 10:37:52,076 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 486, 75.0% complete
2026-03-10 10:40:19,423 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19090024 = 2026-03-10 23:22:00 UTC, 97.4% complete
2026-03-10 10:42:52,077 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 579, 89.4% complete
2026-03-10 10:45:19,424 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19090080 = 2026-03-10 23:59:20 UTC, 100.0% complete
2026-03-10 10:47:52,078 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-10 10:50:19,433 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-03-10 10:50:19,439 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-10 10:50:19,924 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:50:21,298 INFO [download_results] running in process 1246219
2026-03-10 10:50:21,300 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:50:21,301 INFO [download_results] publishing log messages to tcp://*:5562
2026-03-10 10:50:21,302 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:50:21,303 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/10mar26 to /results2/SalishSea/nowcast-green.202111
2026-03-10 10:52:52,082 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-03-10 10:52:52,086 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-03-10 10:52:52,589 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:52:54,110 INFO [download_wwatch3_results] running in process 1246273
2026-03-10 10:52:54,112 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:52:54,113 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2026-03-10 10:52:54,114 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:33,925 INFO [download_results] nowcast-green 2026-03-10 results files from arbutus.cloud-nowcast downloaded
2026-03-10 10:53:33,932 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-10 10:53:34,247 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:53:34,251 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-03-10 10:53:34,257 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:53:34,262 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:53:34,268 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:53:37,395 INFO [download_wwatch3_results] forecast 2026-03-10 results files from arbutus.cloud-nowcast downloaded
2026-03-10 10:53:37,399 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-03-10 10:53:37,449 INFO [ping_erddap] running in process 1246285
2026-03-10 10:53:37,450 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:53:37,451 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-03-10 10:53:37,452 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:37,457 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-03-10 10:53:37,770 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:53:37,777 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 10:53:38,870 INFO [make_averaged_dataset] running in process 1246286
2026-03-10 10:53:38,871 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:53:38,872 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2026-03-10 10:53:38,873 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:38,874 INFO [make_averaged_dataset] creating day-averaged dataset for 10-Mar-2026 physics
2026-03-10 10:53:38,886 INFO [make_averaged_dataset] running in process 1246288
2026-03-10 10:53:38,887 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:53:38,888 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-03-10 10:53:38,889 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:38,890 INFO [make_averaged_dataset] creating day-averaged dataset for 10-Mar-2026 chemistry
2026-03-10 10:53:38,893 INFO [make_averaged_dataset] running in process 1246287
2026-03-10 10:53:38,894 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:53:38,895 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-03-10 10:53:38,896 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:38,897 INFO [make_averaged_dataset] creating day-averaged dataset for 10-Mar-2026 biology
2026-03-10 10:53:38,968 INFO [update_forecast_datasets] running in process 1246338
2026-03-10 10:53:38,969 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:53:38,970 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566
2026-03-10 10:53:38,971 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:38,971 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/10mar26 run
2026-03-10 10:53:39,010 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/10mar26 run
2026-03-10 10:53:39,010 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/10mar26 run
2026-03-10 10:53:39,130 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/10mar26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-03-10 10:53:39,131 INFO [update_forecast_datasets] wwatch3 2026-03-10 forecast rolling forecast datasets updated
2026-03-10 10:53:39,134 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-10 10:53:39,499 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-03-10 10:53:40,282 INFO [make_plots] running in process 1246283
2026-03-10 10:53:40,283 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:53:40,284 INFO [make_plots] publishing log messages to tcp://*:5566
2026-03-10 10:53:40,285 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:40,658 INFO [make_plots] preparing render list for 2026-03-10 NEMO nowcast-green research figures
2026-03-10 10:53:40,735 INFO [ping_erddap] running in process 1246364
2026-03-10 10:53:40,736 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:53:40,737 INFO [ping_erddap] publishing log messages to tcp://*:5567
2026-03-10 10:53:40,755 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:40,781 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-03-10 10:53:40,783 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-10 10:53:41,047 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-03-10'], host='localhost')
2026-03-10 10:53:42,518 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:43,303 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:44,052 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:44,167 INFO [make_plots] running in process 1246392
2026-03-10 10:53:44,168 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 10:53:44,169 INFO [make_plots] publishing log messages to tcp://*:5567
2026-03-10 10:53:44,170 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-10 10:53:44,171 INFO [make_plots] preparing render list for 2026-03-10 WaveWatch3 forecast publish figures
2026-03-10 10:53:44,172 INFO [make_plots] added Halibut Bank figure to 2026-03-10 WaveWatch3 forecast publish render list
2026-03-10 10:53:44,172 INFO [make_plots] added Sentry Shoal figure to 2026-03-10 WaveWatch3 forecast publish render list
2026-03-10 10:53:44,173 INFO [make_plots] starting to render wwatch3 forecast publish 10mar26 figures
2026-03-10 10:53:44,795 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:44,844 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-03-10 10:53:45,542 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:46,354 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:47,167 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:48,106 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:49,036 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:49,978 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:50,911 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:51,698 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:51,891 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-03-10 10:53:52,848 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:52,849 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:52,850 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:52,851 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-03-10 NEMO nowcast-green research render list
2026-03-10 10:53:52,852 INFO [make_plots] starting to render nemo nowcast-green research 10mar26 figures
2026-03-10 10:53:56,190 INFO [make_plots] finished rendering wwatch3 forecast publish 10mar26 figures
2026-03-10 10:53:56,199 INFO [make_plots] wwatch3 publish plots for 2026-03-10 forecast completed
2026-03-10 10:53:56,218 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-10 10:54:38,427 INFO [make_averaged_dataset] day-averaged dataset for 10-Mar-2026 physics created
2026-03-10 10:54:38,430 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-10 10:54:41,498 INFO [make_averaged_dataset] day-averaged dataset for 10-Mar-2026 chemistry created
2026-03-10 10:54:41,500 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-10 10:54:50,744 INFO [make_averaged_dataset] day-averaged dataset for 10-Mar-2026 biology created
2026-03-10 10:54:50,747 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-03-10 11:59:22,668 INFO [make_plots] finished rendering nemo nowcast-green research 10mar26 figures
2026-03-10 11:59:22,678 INFO [make_plots] nemo research plots for 2026-03-10 nowcast-green completed
2026-03-10 11:59:22,746 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-10 14:42:20,212 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260310/18/
2026-03-10 14:42:20,213 INFO [collect_weather] 2026-03-10 2.5km weather forecast 18 collection complete
2026-03-10 14:42:20,216 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-10 14:42:20,796 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-03-10 14:42:20,808 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-03-10 14:42:20,814 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-03-10 14:42:20,825 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-03-11'], host='localhost')
2026-03-10 14:42:22,049 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260310/18/
2026-03-10 14:42:22,050 INFO [crop_gribs] 2026-03-10 18 GRIBs cropping complete
2026-03-10 14:42:22,055 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-10 14:42:23,122 INFO [collect_weather] running in process 1249233
2026-03-10 14:42:23,124 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 14:42:23,124 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-10 14:42:23,125 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-10 14:42:23,126 INFO [download_weather] running in process 1249230
2026-03-10 14:42:23,127 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 14:42:23,128 INFO [download_weather] publishing log messages to tcp://*:5563
2026-03-10 14:42:23,129 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-10 14:42:23,130 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260310
2026-03-10 14:42:23,131 INFO [download_weather] running in process 1249232
2026-03-10 14:42:23,132 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 14:42:23,133 INFO [download_weather] publishing log messages to tcp://*:5564
2026-03-10 14:42:23,134 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-10 14:42:23,135 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260310
2026-03-10 14:42:23,221 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-03-10 14:42:24,684 INFO [crop_gribs] running in process 1249234
2026-03-10 14:42:24,685 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 14:42:24,686 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-03-10 14:42:24,687 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-10 14:42:24,688 INFO [crop_gribs] cropping 2026-03-11 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-03-10 14:42:24,701 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260311/00/
2026-03-10 14:45:57,406 INFO [download_weather] 2026-03-10 1km weather forecast 00 downloads complete
2026-03-10 14:45:57,409 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-10 14:45:59,260 INFO [download_weather] 2026-03-10 1km weather forecast 12 downloads complete
2026-03-10 14:45:59,263 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-03-10 20:45:11,832 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260311/00/
2026-03-10 20:45:11,834 INFO [collect_weather] 2026-03-11 2.5km weather forecast 00 collection complete
2026-03-10 20:45:11,837 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-10 20:45:12,368 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-03-10 20:45:12,376 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-03-11'], host='localhost')
2026-03-10 20:45:13,572 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260311/00/
2026-03-10 20:45:13,573 INFO [crop_gribs] 2026-03-11 00 GRIBs cropping complete
2026-03-10 20:45:13,577 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-10 20:45:14,600 INFO [collect_weather] running in process 1253383
2026-03-10 20:45:14,602 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 20:45:14,603 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-10 20:45:14,603 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-10 20:45:14,624 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-03-10 20:45:16,002 INFO [crop_gribs] running in process 1253385
2026-03-10 20:45:16,003 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-10 20:45:16,004 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-03-10 20:45:16,005 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-10 20:45:16,006 INFO [crop_gribs] cropping 2026-03-11 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-03-10 20:45:16,023 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260311/06/
2026-03-11 02:41:07,241 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260311/06/
2026-03-11 02:41:07,241 INFO [collect_weather] 2026-03-11 2.5km weather forecast 06 collection complete
2026-03-11 02:41:07,244 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-03-11 02:41:07,813 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,821 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,826 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,830 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,834 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,838 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,845 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,854 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,860 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,864 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,871 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,876 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,882 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,890 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,903 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-03-11 02:41:07,915 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-03-11 02:41:07,938 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-03-11 02:41:07,945 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-03-10'], host='localhost')
2026-03-11 02:41:07,953 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-03-11 02:41:07,959 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-03-11 02:41:07,977 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-03-11 02:41:08,229 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260311/06/
2026-03-11 02:41:08,231 INFO [crop_gribs] 2026-03-11 06 GRIBs cropping complete
2026-03-11 02:41:08,246 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-03-11 02:41:09,513 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-03-11 02:41:11,109 INFO [collect_NeahBay_ssh] running in process 1257297
2026-03-11 02:41:11,110 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:11,122 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-03-11 02:41:11,130 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:11,132 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-03-11
2026-03-11 02:41:11,134 INFO [collect_weather] running in process 1257298
2026-03-11 02:41:11,135 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:11,139 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-03-11 02:41:11,139 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:11,161 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-03-11 02:41:12,072 INFO [collect_river_data] running in process 1257278
2026-03-11 02:41:12,073 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,075 INFO [collect_river_data] publishing log messages to tcp://*:5564
2026-03-11 02:41:12,076 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,078 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-03-10
2026-03-11 02:41:12,151 INFO [collect_river_data] running in process 1257292
2026-03-11 02:41:12,152 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,152 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-03-11 02:41:12,153 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,154 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-03-10
2026-03-11 02:41:12,189 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-03-10 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-03-11 02:41:12,189 INFO [collect_river_data] ECCC Capilano river data collection for 2026-03-10 completed
2026-03-11 02:41:12,191 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:12,211 INFO [collect_river_data] running in process 1257280
2026-03-11 02:41:12,216 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,217 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-03-11 02:41:12,218 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,219 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-03-10
2026-03-11 02:41:12,225 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-03-11 02:41:12,226 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-03-10 completed
2026-03-11 02:41:12,268 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-03-11 02:41:12,269 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-03-10 completed
2026-03-11 02:41:12,284 INFO [collect_river_data] running in process 1257288
2026-03-11 02:41:12,296 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,304 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-03-11 02:41:12,312 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,320 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-03-10
2026-03-11 02:41:12,321 INFO [collect_river_data] running in process 1257286
2026-03-11 02:41:12,322 INFO [collect_river_data] running in process 1257285
2026-03-11 02:41:12,324 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,327 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,329 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-03-11 02:41:12,330 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-03-11 02:41:12,332 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,333 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,336 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-03-10
2026-03-11 02:41:12,337 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-03-10
2026-03-11 02:41:12,339 INFO [collect_river_data] running in process 1257289
2026-03-11 02:41:12,340 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,342 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-03-11 02:41:12,345 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,346 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-03-10
2026-03-11 02:41:12,350 INFO [collect_river_data] running in process 1257291
2026-03-11 02:41:12,353 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,354 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-03-11 02:41:12,356 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,357 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-03-10
2026-03-11 02:41:12,361 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-03-11 02:41:12,362 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-03-10 completed
2026-03-11 02:41:12,374 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-03-11 02:41:12,375 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-03-10 completed
2026-03-11 02:41:12,381 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-03-11 02:41:12,381 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-03-10 completed
2026-03-11 02:41:12,383 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-03-11 02:41:12,383 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-03-10 completed
2026-03-11 02:41:12,397 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-03-11 02:41:12,398 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-03-10 completed
2026-03-11 02:41:12,399 INFO [collect_river_data] running in process 1257287
2026-03-11 02:41:12,399 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,400 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-03-11 02:41:12,405 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,406 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-03-10
2026-03-11 02:41:12,424 INFO [collect_river_data] running in process 1257283
2026-03-11 02:41:12,424 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,425 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-03-11 02:41:12,426 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,427 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-03-10
2026-03-11 02:41:12,431 INFO [collect_river_data] running in process 1257282
2026-03-11 02:41:12,433 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,434 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-03-11 02:41:12,436 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,437 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-03-10
2026-03-11 02:41:12,444 INFO [collect_river_data] running in process 1257290
2026-03-11 02:41:12,445 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,446 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-03-11 02:41:12,447 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,448 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-03-10
2026-03-11 02:41:12,449 INFO [collect_river_data] running in process 1257284
2026-03-11 02:41:12,450 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,452 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-03-11 02:41:12,453 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,456 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-03-10
2026-03-11 02:41:12,457 INFO [collect_river_data] running in process 1257281
2026-03-11 02:41:12,459 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,461 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-03-11 02:41:12,462 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-03-10 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-03-11 02:41:12,463 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,464 INFO [collect_river_data] ECCC Fraser river data collection for 2026-03-10 completed
2026-03-11 02:41:12,466 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-03-10
2026-03-11 02:41:12,471 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-03-10 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-03-11 02:41:12,473 INFO [collect_river_data] ECCC Englishman river data collection for 2026-03-10 completed
2026-03-11 02:41:12,484 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-03-11 02:41:12,488 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-03-10 completed
2026-03-11 02:41:12,497 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-03-11 02:41:12,498 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-03-10 completed
2026-03-11 02:41:12,506 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-03-11 02:41:12,507 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-03-10 completed
2026-03-11 02:41:12,519 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-03-10 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-03-11 02:41:12,520 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-03-10 completed
2026-03-11 02:41:12,673 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:12,722 INFO [get_vfpa_hadcp] running in process 1257296
2026-03-11 02:41:12,725 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,726 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2026-03-11 02:41:12,728 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,731 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-10
2026-03-11 02:41:12,880 INFO [crop_gribs] running in process 1257299
2026-03-11 02:41:12,881 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:12,882 INFO [crop_gribs] publishing log messages to tcp://*:5579
2026-03-11 02:41:12,884 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:12,884 INFO [crop_gribs] cropping 2026-03-11 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-03-11 02:41:12,900 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260311/12/
2026-03-11 02:41:13,169 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:13,249 INFO [grib_to_netcdf] running in process 1257519
2026-03-11 02:41:13,250 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:13,251 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2026-03-11 02:41:13,252 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:13,252 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-03-11 forecast2 run
2026-03-11 02:41:13,293 INFO [collect_NeahBay_ssh] 2026-03-11T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-03-11 02:41:13,535 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:13,743 INFO [get_onc_ferry] running in process 1257295
2026-03-11 02:41:13,744 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:13,745 INFO [get_onc_ferry] publishing log messages to tcp://*:5565
2026-03-11 02:41:13,745 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:13,746 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-03-10
2026-03-11 02:41:13,830 INFO [get_onc_ctd] running in process 1257293
2026-03-11 02:41:13,832 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:13,833 INFO [get_onc_ctd] publishing log messages to tcp://*:5566
2026-03-11 02:41:13,834 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:13,835 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-03-10
2026-03-11 02:41:13,885 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:13,890 INFO [get_onc_ctd] running in process 1257294
2026-03-11 02:41:13,892 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:13,893 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2026-03-11 02:41:13,893 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:13,894 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-03-10
2026-03-11 02:41:14,230 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:14,571 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:14,894 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:15,311 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:15,524 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-03-10 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202603.nc
2026-03-11 02:41:15,525 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-03 netcdf file
2026-03-11 02:41:15,635 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:15,931 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:16,229 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:16,562 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:16,994 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-03-11 02:41:17,326 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-03-11 02:41:17,648 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 02:41:17,654 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2026-03-11 02:41:18,075 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2026-03-11 02:41:19,358 INFO [ping_erddap] running in process 1257800
2026-03-11 02:41:19,359 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:19,360 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-11 02:41:19,361 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:19,362 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2026-03-11 02:41:19,364 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 02:41:21,359 INFO [make_ssh_files] running in process 1257787
2026-03-11 02:41:21,361 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:21,362 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-03-11 02:41:21,363 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:21,364 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-03-11 NOAA Neah Bay 00Z observation and forecast values
2026-03-11 02:41:23,546 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d06.nc
2026-03-11 02:41:23,753 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d07.nc
2026-03-11 02:41:23,951 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d08.nc
2026-03-11 02:41:24,160 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d09.nc
2026-03-11 02:41:24,366 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m03d10.nc
2026-03-11 02:41:24,601 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d11.nc
2026-03-11 02:41:24,836 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d12.nc
2026-03-11 02:41:25,055 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d13.nc
2026-03-11 02:41:25,177 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d12.nc
2026-03-11 02:41:25,279 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m03d14.nc
2026-03-11 02:41:25,509 INFO [make_ssh_files] sea surface height boundary file for 2026-03-11 forecast2 run created
2026-03-11 02:41:25,513 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-03-11 02:41:25,847 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-03-11 02:41:25,852 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2026-03-11 02:41:27,341 INFO [make_201702_runoff_file] running in process 1257859
2026-03-11 02:41:27,342 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:27,343 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2026-03-11 02:41:27,344 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:27,374 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2026-03-11 02:41:27,502 INFO [make_201702_runoff_file] 2026-03-11 runoff file creation from Fraser at Hope and climatology elsewhere complete
2026-03-11 02:41:27,504 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2026-03-11 02:41:28,110 INFO [make_runoff_file] running in process 1257857
2026-03-11 02:41:28,111 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:28,112 INFO [make_runoff_file] publishing log messages to tcp://*:5567
2026-03-11 02:41:28,113 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:28,116 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-03-10
2026-03-11 02:41:28,591 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m03d13.nc
2026-03-11 02:41:28,598 INFO [grib_to_netcdf] 2026-03-11 NEMO-atmos forcing file for forecast2 created
2026-03-11 02:41:28,600 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-03-11 02:41:29,143 ERROR [make_runoff_file] no 2026-03-10 discharge obs for SkagitMountVernon: patching
2026-03-11 02:41:29,787 ERROR [make_runoff_file] no 2026-03-10 discharge obs for SnohomishMonroe: patching
2026-03-11 02:41:30,022 ERROR [make_runoff_file] no 2026-03-10 discharge obs for NisquallyMcKenna: patching
2026-03-11 02:41:30,135 ERROR [make_runoff_file] no 2026-03-10 discharge obs for GreenwaterGreenwater: patching
2026-03-11 02:41:31,556 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-03-10: /results/forcing/rivers/R202108Dailies_y2026m03d10.nc
2026-03-11 02:41:31,558 INFO [make_runoff_file] 2026-03-10 runoff file creation completed
2026-03-11 02:41:31,560 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-03-11 02:41:31,889 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-03-11 02:41:31,893 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2026-03-11 02:41:31,898 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-03-11 02:41:31,904 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-03-11 02:41:32,403 INFO [get_onc_ctd] 2026-03-10 ONC SEVIP CTD T&S file created
2026-03-11 02:41:32,405 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-11 02:41:32,811 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-03-11 02:41:33,443 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-03-10
2026-03-11 02:41:33,497 INFO [upload_forcing] running in process 1257905
2026-03-11 02:41:33,498 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:33,499 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-03-11 02:41:33,500 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:33,598 INFO [upload_forcing] running in process 1257906
2026-03-11 02:41:33,599 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:33,600 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-03-11 02:41:33,601 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:33,605 INFO [upload_forcing] running in process 1257904
2026-03-11 02:41:33,605 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:33,606 INFO [upload_forcing] publishing log messages to tcp://*:5567
2026-03-11 02:41:33,607 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:33,639 INFO [upload_forcing] running in process 1257907
2026-03-11 02:41:33,640 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:33,641 INFO [upload_forcing] publishing log messages to tcp://*:5568
2026-03-11 02:41:33,642 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:33,925 INFO [get_onc_ctd] 2026-03-10 ONC SCVIP CTD T&S file created
2026-03-11 02:41:33,928 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-03-11 02:41:34,006 INFO [ping_erddap] running in process 1257911
2026-03-11 02:41:34,007 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:34,008 INFO [ping_erddap] publishing log messages to tcp://*:5569
2026-03-11 02:41:34,009 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:34,010 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-03-11 02:41:34,273 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-03-11 02:41:34,279 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 02:41:35,049 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m03d10.nc
2026-03-11 02:41:35,090 INFO [upload_forcing] forecast2 2026-03-11 forcing files upload to optimum-hindcast completed
2026-03-11 02:41:35,093 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 02:41:35,559 INFO [ping_erddap] running in process 1257939
2026-03-11 02:41:35,560 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:35,561 INFO [ping_erddap] publishing log messages to tcp://*:5566
2026-03-11 02:41:35,562 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:35,564 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-03-11 02:41:35,566 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 02:41:36,313 INFO [upload_forcing] forecast2 2026-03-11 forcing files upload to arbutus.cloud-nowcast completed
2026-03-11 02:41:36,317 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 02:41:36,640 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-11'], host='localhost')
2026-03-11 02:41:37,996 INFO [make_forcing_links] running in process 1257950
2026-03-11 02:41:37,998 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:37,999 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2026-03-11 02:41:38,000 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:39,010 INFO [upload_forcing] forecast2 2026-03-11 forcing files upload to orcinus-nowcast-agrif completed
2026-03-11 02:41:39,014 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 02:41:39,122 INFO [make_forcing_links] forecast2 2026-03-11 forcing file links on arbutus.cloud-nowcast created
2026-03-11 02:41:39,339 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-03-11 02:41:39,629 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 02:41:41,737 INFO [run_NEMO] running in process 25392
2026-03-11 02:41:41,741 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:41,742 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-03-11 02:41:41,801 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:42,599 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2026-03-11 02:41:42,912 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/10mar26forecast2_2026-03-11T024141.876478-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-03-11 02:41:42,926 INFO [run_NEMO] forecast2 NEMO run for 2026-03-11 on arbutus.cloud-nowcast started
2026-03-11 02:41:42,930 INFO [upload_forcing] forecast2 2026-03-11 forcing files upload to robot.nibi completed
2026-03-11 02:41:42,932 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-03-11 02:41:43,267 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-11 02:41:43,274 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-03-11 02:41:45,275 INFO [watch_NEMO] running in process 25665
2026-03-11 02:41:45,276 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:41:45,277 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-03-11 02:41:45,278 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:41:45,343 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-03-11 02:41:55,977 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-03-10
2026-03-11 02:42:17,247 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-03-10
2026-03-11 02:42:38,567 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-03-10
2026-03-11 02:42:38,628 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,634 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,636 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,637 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-03-10
2026-03-11 02:42:38,701 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,706 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,708 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,709 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-03-10
2026-03-11 02:42:38,776 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,779 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,780 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-03-10
2026-03-11 02:42:38,845 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,848 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,849 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-03-10
2026-03-11 02:42:38,917 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:38,920 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-10; substituting empty dataset
2026-03-11 02:42:39,116 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-03-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-11 02:42:39,131 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-03-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-11 02:42:39,146 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-03-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-11 02:42:39,161 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-03-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-11 02:42:39,175 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-03-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-11 02:42:39,191 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-03-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-11 02:42:39,206 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-03-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-03-11 02:43:15,183 INFO [get_onc_ferry] 2026-03-10 ONC TWDP ferry data file created
2026-03-11 02:43:15,185 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-03-11 02:43:15,458 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-03-11 02:43:16,756 INFO [ping_erddap] running in process 1258010
2026-03-11 02:43:16,758 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 02:43:16,759 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-11 02:43:16,760 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 02:43:16,762 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-03-11 02:43:16,764 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 02:46:45,344 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19092730 = 2026-03-12 05:26:00 UTC, 18.1% complete
2026-03-11 02:51:45,345 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19093299 = 2026-03-12 11:45:20 UTC, 39.2% complete
2026-03-11 02:56:45,350 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19093871 = 2026-03-12 18:06:40 UTC, 60.4% complete
2026-03-11 03:01:45,347 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19094402 = 2026-03-13 00:00:40 UTC, 80.1% complete
2026-03-11 03:06:45,347 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19094940 = 2026-03-13 05:59:20 UTC, 100.0% complete
2026-03-11 03:11:45,352 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-03-11 03:11:45,357 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-03-11 03:11:45,886 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 03:11:45,900 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 03:11:45,905 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-11 03:11:47,141 INFO [download_results] running in process 1259270
2026-03-11 03:11:47,142 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:11:47,143 INFO [download_results] publishing log messages to tcp://*:5563
2026-03-11 03:11:47,144 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:11:47,145 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/10mar26 to /results/SalishSea/forecast2.202111
2026-03-11 03:11:48,623 INFO [make_ww3_current_file] running in process 25953
2026-03-11 03:11:48,624 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:11:48,625 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-03-11 03:11:48,627 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:11:48,628 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-03-11 forecast2 run
2026-03-11 03:11:49,692 INFO [make_ww3_wind_file] running in process 26054
2026-03-11 03:11:49,694 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:11:49,695 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-03-11 03:11:49,697 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:11:49,698 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-03-11 forecast2 run
2026-03-11 03:11:58,937 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-03-11 forecast2 run
2026-03-11 03:11:58,943 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-03-11 03:13:16,998 INFO [download_results] forecast2 2026-03-10 results files from arbutus.cloud-nowcast downloaded
2026-03-11 03:13:17,002 INFO [manager] checklist updated with [results files] items from download_results worker
2026-03-11 03:13:17,342 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-11 03:13:19,819 INFO [make_CHS_currents_file] running in process 1259325
2026-03-11 03:13:19,820 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:13:19,821 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-03-11 03:13:19,822 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:13:59,325 INFO [make_CHS_currents_file] Made CHS currents file for 2026-03-10 for forecast2
2026-03-11 03:13:59,328 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-03-11 03:13:59,634 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-11 03:14:00,733 INFO [update_forecast_datasets] running in process 1259368
2026-03-11 03:14:00,734 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:14:00,735 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-03-11 03:14:00,736 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:14:00,738 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/10mar26 run
2026-03-11 03:15:25,212 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-03-11 forecast2 run
2026-03-11 03:15:25,214 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-03-11 03:15:25,555 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-03-11'], host='arbutus.cloud-nowcast')
2026-03-11 03:15:28,201 INFO [run_ww3] running in process 26580
2026-03-11 03:15:28,202 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:15:28,203 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-03-11 03:15:28,204 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:15:28,208 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-11T031528.201817-0700
2026-03-11 03:15:28,225 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-03-11 03:15:28,230 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-03-11T031528.201817-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-03-11 03:15:28,232 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-03-11 on arbutus.cloud-nowcast started
2026-03-11 03:15:28,554 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-03-11 03:15:30,486 INFO [watch_ww3] running in process 26669
2026-03-11 03:15:30,487 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:15:30,488 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-03-11 03:15:30,489 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:15:30,535 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-03-11 03:18:18,897 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/10mar26 run: /results/SalishSea/rolling-forecasts/nemo
2026-03-11 03:18:18,898 INFO [update_forecast_datasets] nemo 2026-03-10 forecast2 rolling forecast datasets updated
2026-03-11 03:18:18,901 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-03-11 03:18:19,182 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-03-11 03:18:19,186 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-03-10'], host='localhost')
2026-03-11 03:18:19,192 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-11 03:18:20,627 INFO [ping_erddap] running in process 1259511
2026-03-11 03:18:20,628 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:18:20,629 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-03-11 03:18:20,630 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:18:20,639 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-03-11 03:18:20,641 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-03-11 03:18:22,102 INFO [make_surface_current_tiles] running in process 1259514
2026-03-11 03:18:22,103 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:18:22,104 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-03-11 03:18:22,105 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:18:22,127 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-03-10 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/10mar26
2026-03-11 03:18:23,450 INFO [make_plots] running in process 1259513
2026-03-11 03:18:23,452 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:18:23,454 INFO [make_plots] publishing log messages to tcp://*:5564
2026-03-11 03:18:23,457 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:18:24,090 INFO [make_plots] preparing render list for 2026-03-10 NEMO forecast2 publish figures
2026-03-11 03:18:24,387 INFO [make_plots] added Website_thumbnail figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,389 INFO [make_plots] added Threshold_website figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,390 INFO [make_plots] added PA_tidal_predictions figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,391 INFO [make_plots] added Boundary Bay figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,392 INFO [make_plots] added Campbell River figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,393 INFO [make_plots] added Cherry Point figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,394 INFO [make_plots] added Friday Harbor figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,395 INFO [make_plots] added Halfmoon Bay figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,396 INFO [make_plots] added Nanaimo figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,397 INFO [make_plots] added Neah Bay figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,398 INFO [make_plots] added New Westminster figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,398 INFO [make_plots] added Patricia Bay figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,399 INFO [make_plots] added Point Atkinson figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,400 INFO [make_plots] added Port Renfrew figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,401 INFO [make_plots] added Sand Heads figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,401 INFO [make_plots] added Sandy Cove figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,402 INFO [make_plots] added Squamish figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,403 INFO [make_plots] added Victoria figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,404 INFO [make_plots] added Woodwards Landing figure to 2026-03-10 NEMO forecast2 publish render list
2026-03-11 03:18:24,404 INFO [make_plots] starting to render nemo forecast2 publish 10mar26 figures
2026-03-11 03:20:30,534 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 84, 15.6% complete
2026-03-11 03:25:30,534 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 174, 32.2% complete
2026-03-11 03:25:59,433 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-03-11 03:25:59,434 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-03-11 03:26:22,987 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:27:26,134 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-03-11 03:27:26,135 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-03-11 03:27:47,400 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:28:33,618 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:28:55,330 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:29:14,423 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:29:24,800 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/10mar26
2026-03-11 03:29:31,381 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:29:49,313 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:30:06,933 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:30:23,310 WARNING [make_plots] invalid station number for Squamish station: None
2026-03-11 03:30:23,311 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-03-11 03:30:30,536 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 273, 50.6% complete
2026-03-11 03:30:39,376 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:30:55,584 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-03-06 00:00:00Z to 2026-03-13 05:55:00Z
2026-03-11 03:31:11,784 INFO [make_plots] finished rendering nemo forecast2 publish 10mar26 figures
2026-03-11 03:31:11,787 INFO [make_plots] nemo publish plots for 2026-03-10 forecast2 completed
2026-03-11 03:31:11,793 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-03-11 03:31:12,093 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-03-10'], host='localhost')
2026-03-11 03:31:15,420 INFO [make_feeds] running in process 1264530
2026-03-11 03:31:15,421 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:31:15,423 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-03-11 03:31:15,424 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:31:29,468 INFO [make_feeds] ATOM feeds for 2026-03-10 forecast2 run completed
2026-03-11 03:31:29,471 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-03-11 03:31:29,801 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-03-11 03:31:31,014 INFO [clear_checklist] running in process 1264558
2026-03-11 03:31:31,016 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:31:31,017 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-03-11 03:31:31,018 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:31:31,019 INFO [clear_checklist] requesting that manager clear system state checklist
2026-03-11 03:31:31,021 INFO [manager] writing checklist to log file
2026-03-11 03:31:31,066 INFO [manager] checklist cleared
2026-03-11 03:31:31,068 INFO [clear_checklist] nowcast system checklist cleared
2026-03-11 03:31:31,071 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-03-11 03:31:32,281 INFO [rotate_logs] running in process 1264563
2026-03-11 03:31:32,282 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-03-11 03:31:32,283 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-03-11 03:31:32,284 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-03-11 03:31:32,285 INFO [rotate_logs] rotating log files