2026-09-07 03:58:35,258 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-09-07 03:58:35,267 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-09-07 03:58:35,273 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-09-07 03:58:35,273 INFO [rotate_logs] log files rotated
2026-09-07 03:58:35,278 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-09-07 04:18:24,411 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/06sep26
2026-09-07 04:23:00,562 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/06sep26
2026-09-07 04:23:00,579 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-09-06 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/06sep26
2026-09-07 04:23:00,581 INFO [make_surface_current_tiles] surface current tile figures for 2026-09-06 forecast2 completed
2026-09-07 04:23:00,864 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-09-07 08:26:52,842 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20260907/12/
2026-09-07 08:26:52,843 INFO [collect_weather] 2026-09-07 2.5km weather forecast 12 collection complete
2026-09-07 08:26:52,846 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-09-07 08:26:53,152 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-09-06'], host='localhost')
2026-09-07 08:26:53,160 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-09-06'], host='localhost')
2026-09-07 08:26:53,165 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-09-06'], host='localhost')
2026-09-07 08:26:53,171 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-09-06'], host='localhost')
2026-09-07 08:26:53,175 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-09-07 08:26:53,181 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-09-07 08:26:53,186 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-09-07 08:26:53,191 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-09-07 08:26:53,214 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-09-07 08:26:53,682 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260907/12/
2026-09-07 08:26:53,695 INFO [crop_gribs] 2026-09-07 12 GRIBs cropping complete
2026-09-07 08:26:53,731 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-09-07 08:26:54,044 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-09-07 08:26:55,782 INFO [collect_weather] running in process 3106466
2026-09-07 08:26:55,783 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:55,784 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-09-07 08:26:55,785 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:55,785 INFO [download_live_ocean] running in process 3106465
2026-09-07 08:26:55,786 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:55,787 INFO [download_live_ocean] publishing log messages to tcp://*:5563
2026-09-07 08:26:55,788 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:55,789 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-09-07
2026-09-07 08:26:55,790 INFO [collect_NeahBay_ssh] running in process 3106464
2026-09-07 08:26:55,790 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:55,792 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2026-09-07 08:26:55,792 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:55,793 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-09-07
2026-09-07 08:26:55,801 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-09-07 08:26:56,316 INFO [collect_river_data] running in process 3106458
2026-09-07 08:26:56,317 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:56,318 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-09-07 08:26:56,319 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:56,321 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-09-06
2026-09-07 08:26:56,326 INFO [download_live_ocean] downloaded 10219005 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.09.07/ubc.nc to /results/forcing/LiveOcean/downloaded/20260907/low_passed_UBC.nc
2026-09-07 08:26:56,327 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-09-07 08:26:56,377 INFO [collect_river_data] running in process 3106460
2026-09-07 08:26:56,378 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:56,379 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-09-07 08:26:56,380 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:56,381 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-09-06
2026-09-07 08:26:56,390 INFO [collect_river_data] running in process 3106462
2026-09-07 08:26:56,397 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:56,398 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-09-07 08:26:56,399 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:56,400 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-09-06
2026-09-07 08:26:56,401 INFO [collect_river_data] running in process 3106461
2026-09-07 08:26:56,402 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:56,403 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-09-07 08:26:56,404 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:56,409 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-09-06
2026-09-07 08:26:56,573 INFO [make_turbidity_file] running in process 3106463
2026-09-07 08:26:56,574 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:56,575 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-09-07 08:26:56,576 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:56,577 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-09-07
2026-09-07 08:26:56,803 INFO [grib_to_netcdf] running in process 3106540
2026-09-07 08:26:56,804 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:56,805 INFO [grib_to_netcdf] publishing log messages to tcp://*:5570
2026-09-07 08:26:56,806 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:56,807 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-09-07 nowcast and forecast runs
2026-09-07 08:26:56,855 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-09-06&endDT=2026-09-06 "HTTP/1.1 200 "
2026-09-07 08:26:56,860 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-09-06 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-09-07 08:26:56,861 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-09-06 completed
2026-09-07 08:26:56,863 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-07 08:26:56,869 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-09-06&endDT=2026-09-06 "HTTP/1.1 200 "
2026-09-07 08:26:56,896 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-09-06&endDT=2026-09-06 "HTTP/1.1 200 "
2026-09-07 08:26:56,911 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-09-06&endDT=2026-09-06 "HTTP/1.1 200 "
2026-09-07 08:26:56,929 INFO [collect_NeahBay_ssh] 2026-09-07T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-09-07 08:26:56,999 INFO [crop_gribs] running in process 3106467
2026-09-07 08:26:57,000 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:26:57,001 INFO [crop_gribs] publishing log messages to tcp://*:5571
2026-09-07 08:26:57,003 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:26:57,004 INFO [crop_gribs] cropping 2026-09-07 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-09-07 08:26:57,007 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-09-07 08:26:57,012 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260907/18/
2026-09-07 08:26:57,023 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-09-06 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-09-07 08:26:57,024 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-09-06 completed
2026-09-07 08:26:57,026 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-09-06 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-09-07 08:26:57,027 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-09-06 completed
2026-09-07 08:26:57,043 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-09-06 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-09-07 08:26:57,044 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-09-06 completed
2026-09-07 08:26:57,146 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:26:57,152 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-07 08:26:57,296 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-07 08:26:57,349 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20260907/low_passed_UBC.nc
2026-09-07 08:26:57,350 INFO [download_live_ocean] 2026-09-07 Live Ocean file for Salish Sea western boundary downloaded
2026-09-07 08:26:57,596 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-07 08:26:57,729 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-09-07 08:26:58,020 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:26:59,704 INFO [make_turbidity_file] 2026-09-07 Fraser River turbidity file creation complete
2026-09-07 08:26:59,706 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-09-07 08:27:00,801 INFO [make_ssh_files] running in process 3106601
2026-09-07 08:27:00,802 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:00,803 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-09-07 08:27:00,805 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:00,806 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-09-07 NOAA Neah Bay 06Z observation and forecast values
2026-09-07 08:27:00,839 INFO [make_live_ocean_files] running in process 3106617
2026-09-07 08:27:00,841 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:00,842 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2026-09-07 08:27:00,844 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:00,846 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-09-07 Live Ocean run
2026-09-07 08:27:02,545 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d03.nc
2026-09-07 08:27:02,713 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d04.nc
2026-09-07 08:27:02,869 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d05.nc
2026-09-07 08:27:03,042 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d06.nc
2026-09-07 08:27:03,232 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d07.nc
2026-09-07 08:27:03,412 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d08.nc
2026-09-07 08:27:03,586 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d09.nc
2026-09-07 08:27:03,785 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d10.nc
2026-09-07 08:27:03,918 INFO [make_ssh_files] sea surface height boundary file for 2026-09-07 nowcast run created
2026-09-07 08:27:03,921 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-09-07 08:27:04,088 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-09-07 08:27:06,238 INFO [make_runoff_file] running in process 3106649
2026-09-07 08:27:06,239 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:06,240 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2026-09-07 08:27:06,241 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:06,243 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-09-06
2026-09-07 08:27:08,613 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m09d07.nc
2026-09-07 08:27:09,080 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-09-06: /results/forcing/rivers/R202108Dailies_y2026m09d06.nc
2026-09-07 08:27:09,083 INFO [make_runoff_file] 2026-09-06 runoff file creation completed
2026-09-07 08:27:09,085 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-09-07 08:27:16,607 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m09d08.nc
2026-09-07 08:27:21,222 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m09d09.nc
2026-09-07 08:27:21,236 INFO [grib_to_netcdf] 2026-09-07 NEMO-atmos forcing file for nowcast+ created
2026-09-07 08:27:21,238 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-09-07 08:27:37,269 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m09d07.nc
2026-09-07 08:27:37,269 INFO [make_live_ocean_files] 2026-09-07 Live Ocean western boundary conditions files created
2026-09-07 08:27:37,272 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-09-07 08:27:37,413 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-09-07 08:27:37,420 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-09-07 08:27:37,425 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-09-07 08:27:37,430 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-09-07 08:27:38,900 INFO [ping_erddap] running in process 3106699
2026-09-07 08:27:38,901 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:38,902 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-09-07 08:27:38,903 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:38,904 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-09-07 08:27:38,906 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-07 08:27:39,105 INFO [upload_forcing] running in process 3106702
2026-09-07 08:27:39,106 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:39,107 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-09-07 08:27:39,108 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:39,133 INFO [upload_forcing] running in process 3106703
2026-09-07 08:27:39,134 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:39,136 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-09-07 08:27:39,137 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:39,156 INFO [upload_forcing] running in process 3106701
2026-09-07 08:27:39,157 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:39,158 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-09-07 08:27:39,160 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:40,872 INFO [upload_forcing] nowcast+ 2026-09-07 forcing files upload to optimum-hindcast completed
2026-09-07 08:27:40,876 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-07 08:27:41,517 INFO [upload_forcing] nowcast+ 2026-09-07 forcing files upload to arbutus.cloud-nowcast completed
2026-09-07 08:27:41,521 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-07 08:27:41,666 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:27:43,188 INFO [make_forcing_links] running in process 3106726
2026-09-07 08:27:43,189 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:43,190 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-09-07 08:27:43,192 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:43,726 INFO [make_forcing_links] nowcast+ 2026-09-07 forcing file links on arbutus.cloud-nowcast created
2026-09-07 08:27:43,730 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-09-07 08:27:43,881 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-09-07'], host='arbutus.cloud-nowcast')
2026-09-07 08:27:44,971 INFO [run_NEMO] running in process 856635
2026-09-07 08:27:44,973 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:27:44,974 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-09-07 08:27:44,975 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:27:47,641 INFO [upload_forcing] nowcast+ 2026-09-07 forcing files upload to robot.nibi completed
2026-09-07 08:27:47,646 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-07 08:28:05,012 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/moad_tools
2026-09-07 08:28:05,013 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO-Cmd
2026-09-07 08:28:05,014 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast
2026-09-07 08:28:05,020 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SalishSeaCmd
2026-09-07 08:28:05,030 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/tools
2026-09-07 08:28:05,037 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/07sep26nowcast_2026-09-07T082745.033659-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-09-07 08:28:05,040 INFO [run_NEMO] nowcast NEMO run for 2026-09-07 on arbutus.cloud-nowcast started
2026-09-07 08:28:52,088 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-09-07 08:28:52,234 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-09-07 08:28:53,067 INFO [watch_NEMO] running in process 856795
2026-09-07 08:28:53,068 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:28:53,069 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-09-07 08:28:53,070 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:28:53,168 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19479058 = 2026-09-07 01:58:00 UTC, 8.2% complete
2026-09-07 08:33:53,169 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19480244 = 2026-09-07 15:08:40 UTC, 63.1% complete
2026-09-07 08:38:53,175 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-09-07 08:38:53,180 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-09-07 08:38:53,479 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:38:53,485 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:38:54,931 INFO [download_results] running in process 3107112
2026-09-07 08:38:54,932 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:38:54,934 INFO [download_results] publishing log messages to tcp://*:5563
2026-09-07 08:38:54,935 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:38:54,936 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/07sep26 to /results/SalishSea/nowcast-blue.202111
2026-09-07 08:38:55,020 INFO [make_forcing_links] running in process 3107111
2026-09-07 08:38:55,022 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:38:55,023 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-09-07 08:38:55,024 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:38:56,024 INFO [make_forcing_links] ssh 2026-09-07 forcing file links on arbutus.cloud-nowcast created
2026-09-07 08:38:56,028 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-09-07 08:38:56,175 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-07'], host='arbutus.cloud-nowcast')
2026-09-07 08:38:57,257 INFO [run_NEMO] running in process 857023
2026-09-07 08:38:57,259 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:38:57,260 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-09-07 08:38:57,261 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:38:57,810 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/moad_tools
2026-09-07 08:38:57,812 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO-Cmd
2026-09-07 08:38:57,813 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast
2026-09-07 08:38:58,156 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SalishSeaCmd
2026-09-07 08:38:58,817 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/tools
2026-09-07 08:38:59,254 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/07sep26forecast_2026-09-07T083857.327538-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-09-07 08:38:59,296 INFO [run_NEMO] forecast NEMO run for 2026-09-07 on arbutus.cloud-nowcast started
2026-09-07 08:38:59,301 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-09-07 08:38:59,449 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-09-07 08:39:00,358 INFO [watch_NEMO] running in process 857177
2026-09-07 08:39:00,359 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:39:00,361 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-09-07 08:39:00,362 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:39:00,455 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-09-07 08:40:53,275 INFO [download_results] nowcast 2026-09-07 results files from arbutus.cloud-nowcast downloaded
2026-09-07 08:40:53,280 INFO [manager] checklist updated with [results files] items from download_results worker
2026-09-07 08:40:53,422 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:40:53,428 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-09-06'], host='localhost')
2026-09-07 08:40:53,431 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:40:55,857 INFO [make_CHS_currents_file] running in process 3107164
2026-09-07 08:40:55,858 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:40:55,859 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-09-07 08:40:55,859 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:40:57,628 INFO [make_plots] running in process 3107161
2026-09-07 08:40:57,629 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:40:57,630 INFO [make_plots] publishing log messages to tcp://*:5564
2026-09-07 08:40:57,631 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:40:57,632 INFO [make_plots] running in process 3107163
2026-09-07 08:40:57,633 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:40:57,634 INFO [make_plots] publishing log messages to tcp://*:5565
2026-09-07 08:40:57,635 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:40:57,969 INFO [make_plots] preparing render list for 2026-09-07 NEMO nowcast-blue research figures
2026-09-07 08:40:57,972 INFO [make_plots] preparing render list for 2026-09-06 NEMO nowcast-blue comparison figures
2026-09-07 08:40:58,025 INFO [make_plots] added SH_wind figure to 2026-09-06 NEMO nowcast-blue comparison render list
2026-09-07 08:40:58,026 INFO [make_plots] added Compare_VENUS_East figure to 2026-09-06 NEMO nowcast-blue comparison render list
2026-09-07 08:40:58,027 INFO [make_plots] added Compare_VENUS_Central figure to 2026-09-06 NEMO nowcast-blue comparison render list
2026-09-07 08:40:58,028 INFO [make_plots] starting to render nemo nowcast comparison 06sep26 figures
2026-09-07 08:40:59,124 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-blue research render list
2026-09-07 08:41:00,319 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-blue research render list
2026-09-07 08:41:00,320 INFO [make_plots] added Currents_sections_and_surface figure to 2026-09-07 NEMO nowcast-blue research render list
2026-09-07 08:41:00,321 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-09-07 NEMO nowcast-blue research render list
2026-09-07 08:41:00,321 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-09-07 NEMO nowcast-blue research render list
2026-09-07 08:41:00,322 INFO [make_plots] starting to render nemo nowcast research 07sep26 figures
2026-09-07 08:41:01,076 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:41:01,078 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:41:30,752 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:41:30,755 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:41:30,966 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:41:30,968 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:41:31,583 INFO [make_CHS_currents_file] Made CHS currents file for 2026-09-07 for nowcast
2026-09-07 08:41:31,585 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-09-07 08:42:04,497 INFO [make_plots] finished rendering nemo nowcast research 07sep26 figures
2026-09-07 08:42:04,499 INFO [make_plots] nemo research plots for 2026-09-07 nowcast completed
2026-09-07 08:42:04,518 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-07 08:42:29,120 INFO [make_plots] finished rendering nemo nowcast comparison 06sep26 figures
2026-09-07 08:42:29,123 INFO [make_plots] nemo comparison plots for 2026-09-06 nowcast completed
2026-09-07 08:42:29,127 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-07 08:44:00,460 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19482109 = 2026-09-08 11:52:00 UTC, 33.0% complete
2026-09-07 08:49:00,462 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19483277 = 2026-09-09 00:50:40 UTC, 69.0% complete
2026-09-07 08:54:00,465 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-09-07 08:54:00,468 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-09-07 08:54:00,781 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-07'], host='arbutus.cloud-nowcast')
2026-09-07 08:54:00,794 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-07'], host='arbutus.cloud-nowcast')
2026-09-07 08:54:00,799 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:54:00,806 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:54:00,809 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:54:00,813 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:54:02,015 INFO [make_ww3_current_file] running in process 857359
2026-09-07 08:54:02,016 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:02,017 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-09-07 08:54:02,018 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:02,019 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-09-07 forecast run
2026-09-07 08:54:02,024 INFO [make_ww3_wind_file] running in process 857358
2026-09-07 08:54:02,025 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:02,026 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-09-07 08:54:02,027 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:02,028 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-09-07 forecast run
2026-09-07 08:54:02,825 INFO [download_results] running in process 3107511
2026-09-07 08:54:02,826 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:02,827 INFO [download_results] publishing log messages to tcp://*:5563
2026-09-07 08:54:02,828 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:02,829 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/07sep26 to /results/SalishSea/forecast.202111
2026-09-07 08:54:02,831 INFO [upload_forcing] running in process 3107509
2026-09-07 08:54:02,832 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:02,834 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-09-07 08:54:02,835 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:02,836 INFO [upload_forcing] running in process 3107508
2026-09-07 08:54:02,837 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:02,838 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-09-07 08:54:02,839 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:02,847 INFO [upload_forcing] running in process 3107507
2026-09-07 08:54:02,848 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:02,849 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-09-07 08:54:02,850 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:03,625 INFO [upload_forcing] turbidity 2026-09-07 forcing files upload to arbutus.cloud-nowcast completed
2026-09-07 08:54:03,629 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-07 08:54:03,642 INFO [upload_forcing] turbidity 2026-09-07 forcing files upload to optimum-hindcast completed
2026-09-07 08:54:03,783 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:54:03,790 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-07 08:54:05,365 INFO [make_forcing_links] running in process 3107538
2026-09-07 08:54:05,366 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:05,368 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2026-09-07 08:54:05,369 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:05,862 INFO [make_forcing_links] nowcast-green 2026-09-07 forcing file links on arbutus.cloud-nowcast created
2026-09-07 08:54:05,867 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-09-07 08:54:06,020 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-09-07'], host='arbutus.cloud-nowcast')
2026-09-07 08:54:06,851 INFO [upload_forcing] turbidity 2026-09-07 forcing files upload to robot.nibi completed
2026-09-07 08:54:06,855 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-07 08:54:07,038 INFO [run_NEMO] running in process 857824
2026-09-07 08:54:07,040 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:07,041 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-09-07 08:54:07,042 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:07,528 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/moad_tools
2026-09-07 08:54:07,529 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO-Cmd
2026-09-07 08:54:07,530 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast
2026-09-07 08:54:07,848 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SalishSeaCmd
2026-09-07 08:54:08,416 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/tools
2026-09-07 08:54:08,822 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/07sep26nowcast-green_2026-09-07T085407.077106-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-09-07 08:54:08,866 INFO [run_NEMO] nowcast-green NEMO run for 2026-09-07 on arbutus.cloud-nowcast started
2026-09-07 08:54:08,871 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-09-07 08:54:09,023 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-09-07 08:54:09,346 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-09-07 forecast run
2026-09-07 08:54:09,348 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-09-07 08:54:09,927 INFO [watch_NEMO] running in process 857984
2026-09-07 08:54:09,929 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:54:09,930 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-09-07 08:54:09,932 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:54:10,030 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-09-07 08:55:11,992 INFO [download_results] forecast 2026-09-07 results files from arbutus.cloud-nowcast downloaded
2026-09-07 08:55:11,997 INFO [manager] checklist updated with [results files] items from download_results worker
2026-09-07 08:55:12,125 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:55:15,044 INFO [make_CHS_currents_file] running in process 3107575
2026-09-07 08:55:15,046 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:55:15,047 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-09-07 08:55:15,048 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:55:59,213 INFO [make_CHS_currents_file] Made CHS currents file for 2026-09-07 for forecast
2026-09-07 08:55:59,217 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-09-07 08:55:59,339 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:56:00,923 INFO [update_forecast_datasets] running in process 3107608
2026-09-07 08:56:00,924 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:56:00,925 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-09-07 08:56:00,926 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:56:00,928 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/07sep26 run
2026-09-07 08:56:01,177 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/07sep26 run: /results/SalishSea/rolling-forecasts/nemo
2026-09-07 08:56:01,178 INFO [update_forecast_datasets] nemo 2026-09-07 forecast rolling forecast datasets updated
2026-09-07 08:56:01,181 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-09-07 08:56:01,307 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-09-07 08:56:01,311 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:56:01,317 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 08:56:02,660 INFO [ping_erddap] running in process 3107613
2026-09-07 08:56:02,661 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:56:02,662 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-09-07 08:56:02,663 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:56:02,675 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-09-07 08:56:02,677 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-07 08:56:04,082 INFO [make_surface_current_tiles] running in process 3107616
2026-09-07 08:56:04,083 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:56:04,084 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-09-07 08:56:04,084 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:56:04,087 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-09-07 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/07sep26
2026-09-07 08:56:05,337 INFO [make_plots] running in process 3107615
2026-09-07 08:56:05,338 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:56:05,339 INFO [make_plots] publishing log messages to tcp://*:5564
2026-09-07 08:56:05,340 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:56:05,703 INFO [make_plots] preparing render list for 2026-09-07 NEMO forecast publish figures
2026-09-07 08:56:05,803 INFO [make_plots] added Website_thumbnail figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,804 INFO [make_plots] added Threshold_website figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,805 INFO [make_plots] added PA_tidal_predictions figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,806 INFO [make_plots] added Boundary Bay figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,807 INFO [make_plots] added Campbell River figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,807 INFO [make_plots] added Cherry Point figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,808 INFO [make_plots] added Friday Harbor figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,809 INFO [make_plots] added Halfmoon Bay figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,809 INFO [make_plots] added Nanaimo figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,810 INFO [make_plots] added Neah Bay figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,811 INFO [make_plots] added New Westminster figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,811 INFO [make_plots] added Patricia Bay figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,812 INFO [make_plots] added Point Atkinson figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,812 INFO [make_plots] added Port Renfrew figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,813 INFO [make_plots] added Sand Heads figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,814 INFO [make_plots] added Sandy Cove figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,815 INFO [make_plots] added Squamish figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,815 INFO [make_plots] added Victoria figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,816 INFO [make_plots] added Woodwards Landing figure to 2026-09-07 NEMO forecast publish render list
2026-09-07 08:56:05,817 INFO [make_plots] starting to render nemo forecast publish 07sep26 figures
2026-09-07 08:56:16,561 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-09-07 forecast run
2026-09-07 08:56:16,565 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-09-07 08:56:16,836 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-09-07'], host='arbutus.cloud-nowcast')
2026-09-07 08:56:17,669 INFO [run_ww3] running in process 858021
2026-09-07 08:56:17,670 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:56:17,671 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-09-07 08:56:17,672 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:56:17,674 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-09-07T085617.666420-0700
2026-09-07 08:56:17,675 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-09-07T085617.666420-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-09-07 08:56:17,720 INFO [run_ww3] nowcast WaveWatch3 run for 2026-09-07 on arbutus.cloud-nowcast started
2026-09-07 08:56:17,724 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-09-07 08:56:18,024 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-09-07 08:56:18,844 INFO [watch_ww3] running in process 858040
2026-09-07 08:56:18,845 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 08:56:18,846 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-09-07 08:56:18,847 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-07 08:56:18,908 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-09-07 08:58:56,294 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,296 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,300 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,302 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,610 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,612 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,614 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,615 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,624 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:58:56,626 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 08:59:10,037 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19479320 = 2026-09-07 04:52:40 UTC, 20.3% complete
2026-09-07 09:01:18,912 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 423, 97.9% complete
2026-09-07 09:04:10,039 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19479811 = 2026-09-07 10:20:00 UTC, 43.1% complete
2026-09-07 09:04:12,531 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:04:12,533 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:04:13,853 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-09-07 09:04:13,854 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-09-07 09:04:28,773 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:04:28,775 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:04:32,654 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:05:29,491 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-09-07 09:05:29,492 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-09-07 09:05:49,008 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:06:18,912 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-09-07 09:06:18,918 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-09-07 09:06:19,237 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:06:19,243 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-07'], host='arbutus.cloud-nowcast')
2026-09-07 09:06:20,072 INFO [run_ww3] running in process 858197
2026-09-07 09:06:20,073 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:06:20,074 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-09-07 09:06:20,075 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:06:20,095 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-09-07T090620.069799-0700
2026-09-07 09:06:20,097 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-09-07T090620.069799-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-09-07 09:06:20,146 INFO [run_ww3] forecast WaveWatch3 run for 2026-09-07 on arbutus.cloud-nowcast started
2026-09-07 09:06:20,151 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-09-07 09:06:20,302 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-09-07 09:06:20,880 INFO [download_wwatch3_results] running in process 3109511
2026-09-07 09:06:20,882 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:06:20,883 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2026-09-07 09:06:20,884 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:06:21,123 INFO [watch_ww3] running in process 858214
2026-09-07 09:06:21,124 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:06:21,125 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-09-07 09:06:21,126 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:06:21,188 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-09-07 09:06:24,921 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:06:45,065 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:06:47,809 INFO [download_wwatch3_results] nowcast 2026-09-07 results files from arbutus.cloud-nowcast downloaded
2026-09-07 09:06:47,812 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-09-07 09:07:05,193 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:07:25,125 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:07:43,077 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:08:02,496 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:08:22,165 WARNING [make_plots] invalid station number for Squamish station: None
2026-09-07 09:08:22,166 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-09-07 09:08:40,815 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:08:44,895 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/07sep26
2026-09-07 09:08:56,933 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:08:57,313 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-09-02 00:00:00Z to 2026-09-09 11:55:00Z
2026-09-07 09:09:10,041 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19480289 = 2026-09-07 15:38:40 UTC, 65.2% complete
2026-09-07 09:09:14,179 INFO [make_plots] finished rendering nemo forecast publish 07sep26 figures
2026-09-07 09:09:14,181 INFO [make_plots] nemo publish plots for 2026-09-07 forecast completed
2026-09-07 09:09:14,188 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-07 09:09:14,343 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:09:17,955 INFO [make_feeds] running in process 3109888
2026-09-07 09:09:17,956 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:09:17,957 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-09-07 09:09:17,958 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:09:32,920 INFO [make_feeds] ATOM feeds for 2026-09-07 forecast run completed
2026-09-07 09:09:32,923 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-09-07 09:11:21,193 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 414, 63.9% complete
2026-09-07 09:13:43,653 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/07sep26
2026-09-07 09:13:43,672 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-09-07 forecast into /results/nowcast-sys/figures/surface_currents/forecast/07sep26
2026-09-07 09:13:43,673 INFO [make_surface_current_tiles] surface current tile figures for 2026-09-07 forecast completed
2026-09-07 09:13:44,134 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-09-07 09:14:10,043 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19480771 = 2026-09-07 21:00:00 UTC, 87.5% complete
2026-09-07 09:16:21,194 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-09-07 09:16:21,199 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-09-07 09:16:21,638 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:16:23,071 INFO [download_wwatch3_results] running in process 3110115
2026-09-07 09:16:23,074 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:16:23,076 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2026-09-07 09:16:23,077 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:16:55,958 INFO [download_wwatch3_results] forecast 2026-09-07 results files from arbutus.cloud-nowcast downloaded
2026-09-07 09:16:55,962 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-09-07 09:16:56,262 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:16:57,538 INFO [update_forecast_datasets] running in process 3110133
2026-09-07 09:16:57,540 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:16:57,541 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-09-07 09:16:57,542 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:16:57,543 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/07sep26 run
2026-09-07 09:16:57,546 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/07sep26 run
2026-09-07 09:16:57,547 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/07sep26 run
2026-09-07 09:16:57,623 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/07sep26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-09-07 09:16:57,624 INFO [update_forecast_datasets] wwatch3 2026-09-07 forecast rolling forecast datasets updated
2026-09-07 09:16:57,628 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-09-07 09:16:57,907 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-09-07 09:16:59,159 INFO [ping_erddap] running in process 3110139
2026-09-07 09:16:59,160 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:16:59,162 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-09-07 09:16:59,163 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:16:59,165 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-09-07 09:16:59,167 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-07 09:16:59,463 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:17:03,001 INFO [make_plots] running in process 3110145
2026-09-07 09:17:03,002 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:17:03,003 INFO [make_plots] publishing log messages to tcp://*:5563
2026-09-07 09:17:03,004 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:17:03,005 INFO [make_plots] preparing render list for 2026-09-07 WaveWatch3 forecast publish figures
2026-09-07 09:17:03,006 INFO [make_plots] added Halibut Bank figure to 2026-09-07 WaveWatch3 forecast publish render list
2026-09-07 09:17:03,007 INFO [make_plots] added Sentry Shoal figure to 2026-09-07 WaveWatch3 forecast publish render list
2026-09-07 09:17:03,008 INFO [make_plots] starting to render wwatch3 forecast publish 07sep26 figures
2026-09-07 09:17:03,406 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-09-07 09:17:08,460 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:17:08,462 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:17:08,486 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:17:08,488 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:17:09,019 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-09-07 09:17:14,395 INFO [make_plots] finished rendering wwatch3 forecast publish 07sep26 figures
2026-09-07 09:17:14,396 INFO [make_plots] wwatch3 publish plots for 2026-09-07 forecast completed
2026-09-07 09:17:14,400 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-07 09:19:10,053 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-09-07 09:19:10,057 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-09-07 09:19:10,497 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:19:11,936 INFO [download_results] running in process 3110210
2026-09-07 09:19:11,937 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:19:11,939 INFO [download_results] publishing log messages to tcp://*:5563
2026-09-07 09:19:11,940 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:19:11,941 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/07sep26 to /results2/SalishSea/nowcast-green.202111
2026-09-07 09:23:50,000 INFO [download_results] nowcast-green 2026-09-07 results files from arbutus.cloud-nowcast downloaded
2026-09-07 09:23:50,007 INFO [manager] checklist updated with [results files] items from download_results worker
2026-09-07 09:23:50,254 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:23:50,260 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-09-07 09:23:50,264 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:23:50,269 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:23:50,273 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-09-07'], host='localhost')
2026-09-07 09:23:52,050 INFO [ping_erddap] running in process 3110305
2026-09-07 09:23:52,050 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:23:52,051 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-09-07 09:23:52,052 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:23:52,056 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-09-07 09:23:52,057 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-07 09:23:53,269 INFO [make_averaged_dataset] running in process 3110306
2026-09-07 09:23:53,271 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:23:53,271 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-09-07 09:23:53,273 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:23:53,274 INFO [make_averaged_dataset] creating day-averaged dataset for 07-Sep-2026 biology
2026-09-07 09:23:53,274 INFO [make_averaged_dataset] running in process 3110307
2026-09-07 09:23:53,275 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:23:53,276 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2026-09-07 09:23:53,276 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:23:53,277 INFO [make_averaged_dataset] creating day-averaged dataset for 07-Sep-2026 physics
2026-09-07 09:23:53,278 INFO [make_averaged_dataset] running in process 3110308
2026-09-07 09:23:53,279 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:23:53,279 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-09-07 09:23:53,280 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:23:53,281 INFO [make_averaged_dataset] creating day-averaged dataset for 07-Sep-2026 chemistry
2026-09-07 09:23:54,634 INFO [make_plots] running in process 3110303
2026-09-07 09:23:54,635 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 09:23:54,636 INFO [make_plots] publishing log messages to tcp://*:5566
2026-09-07 09:23:54,637 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-07 09:23:54,986 INFO [make_plots] preparing render list for 2026-09-07 NEMO nowcast-green research figures
2026-09-07 09:23:56,705 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:23:57,524 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:23:58,302 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:23:59,088 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:23:59,853 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:00,602 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:01,350 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:02,234 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:03,129 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:03,910 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:04,711 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:05,420 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:06,493 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:06,494 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:06,495 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:06,495 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-09-07 NEMO nowcast-green research render list
2026-09-07 09:24:06,496 INFO [make_plots] starting to render nemo nowcast-green research 07sep26 figures
2026-09-07 09:24:07,178 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:24:07,180 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:24:38,964 INFO [make_averaged_dataset] day-averaged dataset for 07-Sep-2026 chemistry created
2026-09-07 09:24:38,968 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-09-07 09:24:49,207 INFO [make_averaged_dataset] day-averaged dataset for 07-Sep-2026 physics created
2026-09-07 09:24:49,210 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-09-07 09:25:01,572 INFO [make_averaged_dataset] day-averaged dataset for 07-Sep-2026 biology created
2026-09-07 09:25:01,575 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-09-07 09:45:05,041 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 09:45:05,043 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-07 10:21:13,939 INFO [make_plots] finished rendering nemo nowcast-green research 07sep26 figures
2026-09-07 10:21:14,112 INFO [make_plots] nemo research plots for 2026-09-07 nowcast-green completed
2026-09-07 10:21:14,161 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-07 15:50:33,058 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20260907/18/
2026-09-07 15:50:33,060 INFO [collect_weather] 2026-09-07 2.5km weather forecast 18 collection complete
2026-09-07 15:50:33,070 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-09-07 15:50:33,618 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-09-07 15:50:33,629 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-09-07 15:50:33,635 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-09-07 15:50:33,643 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-09-08'], host='localhost')
2026-09-07 15:50:34,410 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260907/18/
2026-09-07 15:50:34,411 INFO [crop_gribs] 2026-09-07 18 GRIBs cropping complete
2026-09-07 15:50:34,414 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-09-07 15:50:36,460 INFO [collect_weather] running in process 3119570
2026-09-07 15:50:36,462 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 15:50:36,463 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-09-07 15:50:36,464 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-07 15:50:36,466 INFO [download_weather] running in process 3119568
2026-09-07 15:50:36,466 INFO [download_weather] running in process 3119569
2026-09-07 15:50:36,469 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 15:50:36,469 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 15:50:36,470 INFO [download_weather] publishing log messages to tcp://*:5563
2026-09-07 15:50:36,471 INFO [download_weather] publishing log messages to tcp://*:5564
2026-09-07 15:50:36,471 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-07 15:50:36,472 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-07 15:50:36,473 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20260907
2026-09-07 15:50:36,474 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20260907
2026-09-07 15:50:36,513 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-09-07 15:50:37,927 INFO [crop_gribs] running in process 3119571
2026-09-07 15:50:37,928 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 15:50:37,929 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-09-07 15:50:37,931 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-09-07 15:50:37,932 INFO [crop_gribs] cropping 2026-09-08 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-09-07 15:50:37,944 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260908/00/
2026-09-07 15:54:43,479 INFO [download_weather] 2026-09-07 1km weather forecast 00 downloads complete
2026-09-07 15:54:43,483 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-09-07 15:54:46,517 INFO [download_weather] 2026-09-07 1km weather forecast 12 downloads complete
2026-09-07 15:54:46,520 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-09-07 21:32:42,086 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20260908/00/
2026-09-07 21:32:42,087 INFO [collect_weather] 2026-09-08 2.5km weather forecast 00 collection complete
2026-09-07 21:32:42,091 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-09-07 21:32:42,630 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-09-07 21:32:42,637 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-09-08'], host='localhost')
2026-09-07 21:32:43,470 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260908/00/
2026-09-07 21:32:43,472 INFO [crop_gribs] 2026-09-08 00 GRIBs cropping complete
2026-09-07 21:32:43,474 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-09-07 21:32:44,153 INFO [collect_weather] running in process 3127265
2026-09-07 21:32:44,154 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 21:32:44,155 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-09-07 21:32:44,156 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-07 21:32:44,174 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-09-07 21:32:44,898 INFO [crop_gribs] running in process 3127267
2026-09-07 21:32:44,900 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-07 21:32:44,901 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-09-07 21:32:44,902 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-09-07 21:32:44,903 INFO [crop_gribs] cropping 2026-09-08 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-09-07 21:32:44,915 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260908/06/
2026-09-08 03:49:03,990 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20260908/06/
2026-09-08 03:49:03,991 INFO [collect_weather] 2026-09-08 2.5km weather forecast 06 collection complete
2026-09-08 03:49:03,994 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-09-08 03:49:04,530 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,539 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,544 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,549 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,553 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,558 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,563 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,570 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,581 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,589 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,598 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,610 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,625 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,639 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,645 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-09-08 03:49:04,666 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-09-08 03:49:04,705 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-09-08 03:49:04,717 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-09-07'], host='localhost')
2026-09-08 03:49:04,738 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-09-08 03:49:04,749 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-09-08 03:49:04,762 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-09-08 03:49:05,091 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20260908/06/
2026-09-08 03:49:05,092 INFO [crop_gribs] 2026-09-08 06 GRIBs cropping complete
2026-09-08 03:49:05,098 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-09-08 03:49:06,333 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-09-08 03:49:09,075 INFO [collect_NeahBay_ssh] running in process 3135499
2026-09-08 03:49:09,078 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,080 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2026-09-08 03:49:09,081 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:09,082 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-09-08
2026-09-08 03:49:09,169 INFO [collect_weather] running in process 3135500
2026-09-08 03:49:09,170 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,171 INFO [collect_weather] publishing log messages to tcp://*:5563
2026-09-08 03:49:09,173 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:09,200 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-09-08 03:49:09,792 INFO [collect_river_data] running in process 3135494
2026-09-08 03:49:09,794 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,797 INFO [collect_river_data] publishing log messages to tcp://*:5564
2026-09-08 03:49:09,800 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:09,803 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-09-07
2026-09-08 03:49:09,806 INFO [collect_river_data] running in process 3135493
2026-09-08 03:49:09,809 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,811 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-09-08 03:49:09,814 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:09,818 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-09-07
2026-09-08 03:49:09,959 INFO [collect_river_data] running in process 3135480
2026-09-08 03:49:09,962 INFO [collect_river_data] running in process 3135483
2026-09-08 03:49:09,964 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,964 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,965 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-09-08 03:49:09,967 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-09-08 03:49:09,967 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:09,968 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:09,971 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-09-07
2026-09-08 03:49:09,975 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-09-07
2026-09-08 03:49:09,978 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-09-07 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-09-08 03:49:09,980 INFO [collect_river_data] ECCC Capilano river data collection for 2026-09-07 completed
2026-09-08 03:49:09,982 INFO [collect_river_data] running in process 3135485
2026-09-08 03:49:09,984 INFO [collect_river_data] running in process 3135489
2026-09-08 03:49:09,985 INFO [collect_river_data] running in process 3135488
2026-09-08 03:49:09,986 INFO [collect_river_data] running in process 3135487
2026-09-08 03:49:09,987 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,988 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,989 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:09,990 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,991 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:09,993 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-09-08 03:49:09,995 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-09-08 03:49:09,996 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-09-08 03:49:09,998 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-09-08 03:49:09,999 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,001 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,002 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,002 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,003 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-09-07
2026-09-08 03:49:10,005 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-09-07
2026-09-08 03:49:10,005 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-09-07
2026-09-08 03:49:10,006 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-09-07
2026-09-08 03:49:10,010 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-09-07 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-09-08 03:49:10,010 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-09-08 03:49:10,011 INFO [collect_river_data] ECCC Fraser river data collection for 2026-09-07 completed
2026-09-08 03:49:10,012 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-09-07 completed
2026-09-08 03:49:10,014 INFO [collect_river_data] running in process 3135486
2026-09-08 03:49:10,014 INFO [collect_river_data] running in process 3135482
2026-09-08 03:49:10,016 INFO [collect_river_data] running in process 3135484
2026-09-08 03:49:10,021 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:10,023 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:10,025 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:10,026 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-09-08 03:49:10,027 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-09-08 03:49:10,028 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-09-08 03:49:10,029 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-09-08 03:49:10,029 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-09-08 03:49:10,032 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-09-08 03:49:10,033 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-09-08 03:49:10,033 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-09-07 completed
2026-09-08 03:49:10,034 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-09-08 03:49:10,035 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-09-07 completed
2026-09-08 03:49:10,035 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-09-07 completed
2026-09-08 03:49:10,036 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-09-07 completed
2026-09-08 03:49:10,037 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,038 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,038 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,040 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-09-07 completed
2026-09-08 03:49:10,042 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-09-07
2026-09-08 03:49:10,043 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-09-07
2026-09-08 03:49:10,043 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-09-07
2026-09-08 03:49:10,046 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-09-08 03:49:10,047 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-09-07 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-09-08 03:49:10,048 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-09-07 completed
2026-09-08 03:49:10,048 INFO [collect_river_data] ECCC Englishman river data collection for 2026-09-07 completed
2026-09-08 03:49:10,050 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-09-08 03:49:10,051 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-09-07 completed
2026-09-08 03:49:10,054 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-09-08 03:49:10,058 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-09-07 completed
2026-09-08 03:49:10,064 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-09-08 03:49:10,065 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-09-07 completed
2026-09-08 03:49:10,084 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-09-07 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-09-08 03:49:10,085 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-09-07 completed
2026-09-08 03:49:10,356 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:10,380 INFO [get_vfpa_hadcp] running in process 3135498
2026-09-08 03:49:10,381 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:10,382 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2026-09-08 03:49:10,383 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,384 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-09-07
2026-09-08 03:49:10,407 WARNING [get_vfpa_hadcp] file not found: /opp/observations/AISDATA/20260907T000000Z-20260907T010000Z.csv
2026-09-08 03:49:10,418 CRITICAL [get_vfpa_hadcp] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/nemo_nowcast/worker.py", line 388, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 105, in get_vfpa_hadcp
    ds = _make_hour_dataset(csv_dir, data_date, place)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 170, in _make_hour_dataset
    ds = _csv_to_dataset(csv_dir / csv_filename, place)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/get_vfpa_hadcp.py", line 246, in _csv_to_dataset
    df = pandas.read_csv(csv_file, skiprows=3)
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/parsers/readers.py", line 1026, in read_csv
    return _read(filepath_or_buffer, kwds)
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/parsers/readers.py", line 620, in _read
    parser = TextFileReader(filepath_or_buffer, **kwds)
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/parsers/readers.py", line 1620, in __init__
    self._engine = self._make_engine(f, self.engine)
                   ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/parsers/readers.py", line 1880, in _make_engine
    self.handles = get_handle(
                   ~~~~~~~~~~^
        f,
        ^^
    ...<6 lines>...
        storage_options=self.options.get("storage_options", None),
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/SalishSeaNowcast/.pixi/envs/default/lib/python3.14/site-packages/pandas/io/common.py", line 873, in get_handle
    handle = open(
        handle,
    ...<3 lines>...
        newline="",
    )
FileNotFoundError: [Errno 2] No such file or directory: '/opp/observations/AISDATA/20260907T000000Z-20260907T010000Z.csv'
2026-09-08 03:49:10,788 INFO [crop_gribs] running in process 3135501
2026-09-08 03:49:10,789 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:10,792 INFO [crop_gribs] publishing log messages to tcp://*:5579
2026-09-08 03:49:10,792 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:10,793 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,794 INFO [crop_gribs] cropping 2026-09-08 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-09-08 03:49:10,796 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20260908/12/
2026-09-08 03:49:10,796 INFO [grib_to_netcdf] running in process 3135731
2026-09-08 03:49:10,797 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:10,798 INFO [grib_to_netcdf] publishing log messages to tcp://*:5566
2026-09-08 03:49:10,799 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:10,800 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-09-08 forecast2 run
2026-09-08 03:49:10,885 INFO [collect_NeahBay_ssh] 2026-09-08T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-09-08 03:49:11,100 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:11,223 INFO [get_onc_ctd] running in process 3135496
2026-09-08 03:49:11,225 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:11,226 INFO [get_onc_ctd] publishing log messages to tcp://*:5568
2026-09-08 03:49:11,227 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:11,229 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-09-07
2026-09-08 03:49:11,238 INFO [get_onc_ferry] running in process 3135497
2026-09-08 03:49:11,239 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:11,241 INFO [get_onc_ferry] publishing log messages to tcp://*:5590
2026-09-08 03:49:11,241 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:11,242 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-09-07
2026-09-08 03:49:11,271 INFO [get_onc_ctd] running in process 3135495
2026-09-08 03:49:11,272 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:11,273 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2026-09-08 03:49:11,273 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:11,274 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-09-07
2026-09-08 03:49:11,404 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:11,720 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:12,016 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:12,336 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:12,631 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:12,961 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:13,272 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:13,562 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:13,860 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:14,168 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-09-08 03:49:14,482 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-09-08 03:49:14,790 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 03:49:18,672 INFO [make_ssh_files] running in process 3135844
2026-09-08 03:49:18,673 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:18,674 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2026-09-08 03:49:18,675 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:18,676 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-09-08 NOAA Neah Bay 00Z observation and forecast values
2026-09-08 03:49:20,992 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d03.nc
2026-09-08 03:49:21,207 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d04.nc
2026-09-08 03:49:21,384 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d05.nc
2026-09-08 03:49:21,457 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m09d09.nc
2026-09-08 03:49:21,602 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d06.nc
2026-09-08 03:49:21,799 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m09d07.nc
2026-09-08 03:49:22,036 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d08.nc
2026-09-08 03:49:22,237 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d09.nc
2026-09-08 03:49:22,449 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d10.nc
2026-09-08 03:49:22,683 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m09d11.nc
2026-09-08 03:49:22,928 INFO [make_ssh_files] sea surface height boundary file for 2026-09-08 forecast2 run created
2026-09-08 03:49:22,932 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-09-08 03:49:23,287 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-09-08 03:49:24,209 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m09d10.nc
2026-09-08 03:49:24,215 INFO [grib_to_netcdf] 2026-09-08 NEMO-atmos forcing file for forecast2 created
2026-09-08 03:49:24,217 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-09-08 03:49:25,664 INFO [make_runoff_file] running in process 3135883
2026-09-08 03:49:25,668 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:25,669 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2026-09-08 03:49:25,827 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:25,832 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-09-07
2026-09-08 03:49:26,719 ERROR [make_runoff_file] no 2026-09-07 discharge obs for SkagitMountVernon: patching
2026-09-08 03:49:26,851 ERROR [make_runoff_file] no 2026-09-07 discharge obs for SnohomishMonroe: patching
2026-09-08 03:49:27,059 ERROR [make_runoff_file] no 2026-09-07 discharge obs for NisquallyMcKenna: patching
2026-09-08 03:49:27,160 ERROR [make_runoff_file] no 2026-09-07 discharge obs for GreenwaterGreenwater: patching
2026-09-08 03:49:28,473 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-09-07: /results/forcing/rivers/R202108Dailies_y2026m09d07.nc
2026-09-08 03:49:28,474 INFO [make_runoff_file] 2026-09-07 runoff file creation completed
2026-09-08 03:49:28,476 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-09-08 03:49:28,802 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-09-08 03:49:28,806 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-09-08 03:49:28,809 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-09-08 03:49:30,334 INFO [get_onc_ctd] 2026-09-07 ONC SCVIP CTD T&S file created
2026-09-08 03:49:30,337 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-09-08 03:49:30,620 INFO [upload_forcing] running in process 3135908
2026-09-08 03:49:30,621 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:30,623 INFO [upload_forcing] publishing log messages to tcp://*:5562
2026-09-08 03:49:30,624 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:30,635 INFO [upload_forcing] running in process 3135910
2026-09-08 03:49:30,636 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:30,637 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-09-08 03:49:30,638 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:30,648 INFO [upload_forcing] running in process 3135909
2026-09-08 03:49:30,649 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:30,650 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-09-08 03:49:30,651 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:30,668 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-09-08 03:49:31,382 INFO [get_onc_ctd] 2026-09-07 ONC SEVIP CTD T&S file created
2026-09-08 03:49:31,384 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-09-08 03:49:31,727 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-09-08 03:49:31,897 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m09d07.nc
2026-09-08 03:49:31,956 INFO [upload_forcing] forecast2 2026-09-08 forcing files upload to optimum-hindcast completed
2026-09-08 03:49:31,960 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 03:49:32,326 INFO [ping_erddap] running in process 3135922
2026-09-08 03:49:32,327 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:32,327 INFO [ping_erddap] publishing log messages to tcp://*:5566
2026-09-08 03:49:32,328 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:32,329 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-09-08 03:49:32,331 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-08 03:49:32,426 INFO [upload_forcing] forecast2 2026-09-08 forcing files upload to arbutus.cloud-nowcast completed
2026-09-08 03:49:32,660 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 03:49:32,995 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 03:49:33,270 INFO [ping_erddap] running in process 3135934
2026-09-08 03:49:33,271 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:33,272 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-09-08 03:49:33,273 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:33,274 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-09-08 03:49:33,276 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-08 03:49:33,501 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-09-07
2026-09-08 03:49:34,515 INFO [make_forcing_links] running in process 3135942
2026-09-08 03:49:34,516 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:34,517 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-09-08 03:49:34,519 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:35,115 INFO [make_forcing_links] forecast2 2026-09-08 forcing file links on arbutus.cloud-nowcast created
2026-09-08 03:49:35,119 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-09-08 03:49:35,494 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 03:49:36,481 INFO [run_NEMO] running in process 865040
2026-09-08 03:49:36,483 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:49:36,484 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-09-08 03:49:36,485 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:49:37,786 INFO [upload_forcing] forecast2 2026-09-08 forcing files upload to robot.nibi completed
2026-09-08 03:49:37,790 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-09-08 03:49:57,840 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-09-07
2026-09-08 03:49:58,814 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/moad_tools
2026-09-08 03:49:58,815 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO-Cmd
2026-09-08 03:49:58,816 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast
2026-09-08 03:49:59,156 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SalishSeaCmd
2026-09-08 03:49:59,809 WARNING [run_NEMO] revision and status requested for non-existent repo: /nemoShare/MEOPAR/nowcast-sys/tools
2026-09-08 03:50:00,241 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/07sep26forecast2_2026-09-08T034958.353712-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-09-08 03:50:00,285 INFO [run_NEMO] forecast2 NEMO run for 2026-09-08 on arbutus.cloud-nowcast started
2026-09-08 03:50:00,289 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-09-08 03:50:00,604 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-09-08 03:50:01,424 INFO [watch_NEMO] running in process 865196
2026-09-08 03:50:01,426 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:50:01,427 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-09-08 03:50:01,428 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:50:01,522 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-09-08 03:50:23,564 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-09-07
2026-09-08 03:50:49,397 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-09-07
2026-09-08 03:50:49,600 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:49,606 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:49,608 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:49,609 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-09-07
2026-09-08 03:50:49,765 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:49,772 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:49,774 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:49,775 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-09-07
2026-09-08 03:50:49,957 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:49,960 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:49,962 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-09-07
2026-09-08 03:50:50,647 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:50,651 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:50,652 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-09-07
2026-09-08 03:50:50,862 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:50,865 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-09-07; substituting empty dataset
2026-09-08 03:50:51,043 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-09-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-08 03:50:51,057 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-09-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-08 03:50:51,072 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-09-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-08 03:50:51,087 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-09-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-08 03:50:51,101 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-09-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-08 03:50:51,116 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-09-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-08 03:50:51,131 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-09-07 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-09-08 03:51:04,507 INFO [get_onc_ferry] 2026-09-07 ONC TWDP ferry data file created
2026-09-08 03:51:04,510 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-09-08 03:51:04,828 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-09-08 03:51:06,323 INFO [ping_erddap] running in process 3135992
2026-09-08 03:51:06,325 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 03:51:06,325 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-09-08 03:51:06,326 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-08 03:51:06,328 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-09-08 03:51:06,330 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-08 03:55:01,523 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19484285 = 2026-09-09 12:02:40 UTC, 40.2% complete
2026-09-08 04:00:01,525 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19485462 = 2026-09-10 01:07:20 UTC, 83.8% complete
2026-09-08 04:05:01,530 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-09-08 04:05:01,535 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-09-08 04:05:02,143 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 04:05:02,160 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 04:05:02,164 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-07'], host='localhost')
2026-09-08 04:05:03,434 INFO [make_ww3_wind_file] running in process 865403
2026-09-08 04:05:03,436 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:05:03,437 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-09-08 04:05:03,441 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:05:03,443 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-09-08 forecast2 run
2026-09-08 04:05:03,493 INFO [make_ww3_current_file] running in process 865404
2026-09-08 04:05:03,494 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:05:03,495 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-09-08 04:05:03,497 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:05:03,497 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-09-08 forecast2 run
2026-09-08 04:05:03,656 INFO [download_results] running in process 3137232
2026-09-08 04:05:03,657 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:05:03,658 INFO [download_results] publishing log messages to tcp://*:5562
2026-09-08 04:05:03,660 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:05:03,661 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/07sep26 to /results/SalishSea/forecast2.202111
2026-09-08 04:05:12,435 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-09-08 forecast2 run
2026-09-08 04:05:12,440 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-09-08 04:05:36,500 INFO [download_results] forecast2 2026-09-07 results files from arbutus.cloud-nowcast downloaded
2026-09-08 04:05:36,505 INFO [manager] checklist updated with [results files] items from download_results worker
2026-09-08 04:05:36,841 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-09-07'], host='localhost')
2026-09-08 04:05:39,043 INFO [make_CHS_currents_file] running in process 3137252
2026-09-08 04:05:39,045 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:05:39,046 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-09-08 04:05:39,047 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:06:14,035 INFO [make_CHS_currents_file] Made CHS currents file for 2026-09-07 for forecast2
2026-09-08 04:06:14,038 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-09-08 04:06:14,352 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-09-07'], host='localhost')
2026-09-08 04:06:15,883 INFO [update_forecast_datasets] running in process 3137277
2026-09-08 04:06:15,885 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:06:15,886 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-09-08 04:06:15,887 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:06:15,889 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/07sep26 run
2026-09-08 04:07:47,603 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-09-08 forecast2 run
2026-09-08 04:07:47,608 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-09-08 04:07:47,949 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-08'], host='arbutus.cloud-nowcast')
2026-09-08 04:07:48,787 INFO [run_ww3] running in process 865799
2026-09-08 04:07:48,789 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:07:48,790 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-09-08 04:07:48,791 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:07:48,795 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-09-08T040748.788514-0700
2026-09-08 04:07:48,796 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-09-08T040748.788514-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-09-08 04:07:48,838 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-09-08 on arbutus.cloud-nowcast started
2026-09-08 04:07:48,843 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-09-08 04:07:49,158 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-09-08 04:07:49,985 INFO [watch_ww3] running in process 865816
2026-09-08 04:07:49,986 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:07:49,987 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-09-08 04:07:49,988 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:07:50,050 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-09-08 04:10:38,171 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/07sep26 run: /results/SalishSea/rolling-forecasts/nemo
2026-09-08 04:10:38,172 INFO [update_forecast_datasets] nemo 2026-09-07 forecast2 rolling forecast datasets updated
2026-09-08 04:10:38,175 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-09-08 04:10:38,520 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-09-08 04:10:38,527 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-09-07'], host='localhost')
2026-09-08 04:10:38,534 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-09-07'], host='localhost')
2026-09-08 04:10:40,388 INFO [ping_erddap] running in process 3137464
2026-09-08 04:10:40,389 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:10:40,390 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-09-08 04:10:40,391 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:10:40,404 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-09-08 04:10:40,406 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-08 04:10:41,922 INFO [make_surface_current_tiles] running in process 3137467
2026-09-08 04:10:41,923 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:10:41,924 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-09-08 04:10:41,925 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:10:41,960 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-09-07 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/07sep26
2026-09-08 04:10:43,445 INFO [make_plots] running in process 3137466
2026-09-08 04:10:43,446 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:10:43,446 INFO [make_plots] publishing log messages to tcp://*:5564
2026-09-08 04:10:43,447 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:10:43,924 INFO [make_plots] preparing render list for 2026-09-07 NEMO forecast2 publish figures
2026-09-08 04:10:44,051 INFO [make_plots] added Website_thumbnail figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,052 INFO [make_plots] added Threshold_website figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,053 INFO [make_plots] added PA_tidal_predictions figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,054 INFO [make_plots] added Boundary Bay figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,055 INFO [make_plots] added Campbell River figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,055 INFO [make_plots] added Cherry Point figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,056 INFO [make_plots] added Friday Harbor figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,057 INFO [make_plots] added Halfmoon Bay figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,058 INFO [make_plots] added Nanaimo figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,059 INFO [make_plots] added Neah Bay figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,059 INFO [make_plots] added New Westminster figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,060 INFO [make_plots] added Patricia Bay figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,061 INFO [make_plots] added Point Atkinson figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,062 INFO [make_plots] added Port Renfrew figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,063 INFO [make_plots] added Sand Heads figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,064 INFO [make_plots] added Sandy Cove figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,064 INFO [make_plots] added Squamish figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,065 INFO [make_plots] added Victoria figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,066 INFO [make_plots] added Woodwards Landing figure to 2026-09-07 NEMO forecast2 publish render list
2026-09-08 04:10:44,066 INFO [make_plots] starting to render nemo forecast2 publish 07sep26 figures
2026-09-08 04:12:50,053 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 417, 77.2% complete
2026-09-08 04:13:34,246 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,248 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,252 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,253 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,528 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,530 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,532 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,534 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,543 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:13:34,545 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:17:50,054 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-09-08 04:17:50,068 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-09-08 04:17:50,971 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 04:17:52,974 INFO [download_wwatch3_results] running in process 3139028
2026-09-08 04:17:52,975 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:17:52,977 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2026-09-08 04:17:52,978 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:18:02,099 INFO [download_wwatch3_results] forecast2 2026-09-08 results files from arbutus.cloud-nowcast downloaded
2026-09-08 04:18:02,103 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-09-08 04:18:02,398 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-09-08'], host='localhost')
2026-09-08 04:18:03,823 INFO [update_forecast_datasets] running in process 3139043
2026-09-08 04:18:03,824 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:18:03,825 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2026-09-08 04:18:03,826 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:18:03,827 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/08sep26 run
2026-09-08 04:18:03,831 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/08sep26 run
2026-09-08 04:18:03,832 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/08sep26 run
2026-09-08 04:18:30,536 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/08sep26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-09-08 04:18:30,536 INFO [update_forecast_datasets] wwatch3 2026-09-08 forecast2 rolling forecast datasets updated
2026-09-08 04:18:30,539 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-09-08 04:18:31,112 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-09-08 04:18:32,699 INFO [ping_erddap] running in process 3139207
2026-09-08 04:18:32,700 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:18:32,701 INFO [ping_erddap] publishing log messages to tcp://*:5565
2026-09-08 04:18:32,702 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:18:32,703 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-09-08 04:18:32,705 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-09-08 04:18:33,133 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-09-07'], host='localhost')
2026-09-08 04:18:37,810 INFO [make_plots] running in process 3139229
2026-09-08 04:18:37,820 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:18:37,821 INFO [make_plots] publishing log messages to tcp://*:5565
2026-09-08 04:18:37,822 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:18:37,824 INFO [make_plots] preparing render list for 2026-09-07 WaveWatch3 forecast publish figures
2026-09-08 04:18:37,825 INFO [make_plots] added Halibut Bank figure to 2026-09-07 WaveWatch3 forecast publish render list
2026-09-08 04:18:37,826 INFO [make_plots] added Sentry Shoal figure to 2026-09-07 WaveWatch3 forecast publish render list
2026-09-08 04:18:37,827 INFO [make_plots] starting to render wwatch3 forecast publish 07sep26 figures
2026-09-08 04:18:38,877 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-09-08 04:18:44,931 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:18:44,936 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:18:44,969 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:18:44,973 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:18:45,672 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-09-08 04:18:50,939 INFO [make_plots] finished rendering wwatch3 forecast publish 07sep26 figures
2026-09-08 04:18:50,940 INFO [make_plots] wwatch3 publish plots for 2026-09-07 forecast completed
2026-09-08 04:18:50,943 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-08 04:19:05,666 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:19:05,670 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:19:06,828 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-09-08 04:19:06,847 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-09-08 04:19:27,800 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:19:27,802 WARNING [make_plots] findfont: Failed to find font weight medium, now using 400.
2026-09-08 04:19:35,279 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:20:47,907 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-09-08 04:20:47,908 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-09-08 04:21:10,215 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:21:58,893 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:22:04,762 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/07sep26
2026-09-08 04:22:17,059 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:22:33,946 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:22:51,629 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:23:06,900 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:23:24,109 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:23:42,094 WARNING [make_plots] invalid station number for Squamish station: None
2026-09-08 04:23:42,095 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-09-08 04:24:00,200 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:24:15,982 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:24:16,371 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-09-03 00:00:00Z to 2026-09-10 05:55:00Z
2026-09-08 04:24:33,310 INFO [make_plots] finished rendering nemo forecast2 publish 07sep26 figures
2026-09-08 04:24:33,312 INFO [make_plots] nemo publish plots for 2026-09-07 forecast2 completed
2026-09-08 04:24:33,321 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-09-08 04:24:33,674 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-09-07'], host='localhost')
2026-09-08 04:24:37,317 INFO [make_feeds] running in process 3139919
2026-09-08 04:24:37,319 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:24:37,320 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-09-08 04:24:37,321 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:24:52,098 INFO [make_feeds] ATOM feeds for 2026-09-07 forecast2 run completed
2026-09-08 04:24:52,101 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-09-08 04:24:52,426 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-09-08 04:24:53,811 INFO [clear_checklist] running in process 3139941
2026-09-08 04:24:53,812 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:24:53,813 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-09-08 04:24:53,814 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:24:53,815 INFO [clear_checklist] requesting that manager clear system state checklist
2026-09-08 04:24:53,817 INFO [manager] writing checklist to log file
2026-09-08 04:24:53,867 INFO [manager] checklist cleared
2026-09-08 04:24:53,869 INFO [clear_checklist] nowcast system checklist cleared
2026-09-08 04:24:53,871 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-09-08 04:24:55,249 INFO [rotate_logs] running in process 3139948
2026-09-08 04:24:55,250 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-09-08 04:24:55,251 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-09-08 04:24:55,252 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-09-08 04:24:55,253 INFO [rotate_logs] rotating log files