2026-10-14 02:57:42,819 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2026-10-14 02:57:42,828 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2026-10-14 02:57:42,834 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2026-10-14 02:57:42,835 INFO [rotate_logs] log files rotated
2026-10-14 02:57:42,842 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2026-10-14 02:58:38,841 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13oct26
2026-10-14 03:03:46,139 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13oct26
2026-10-14 03:03:46,157 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-10-13 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/13oct26
2026-10-14 03:03:46,158 INFO [make_surface_current_tiles] surface current tile figures for 2026-10-13 forecast2 completed
2026-10-14 03:03:46,599 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-10-14 08:23:51,448 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20261014/12/
2026-10-14 08:23:51,449 INFO [collect_weather] 2026-10-14 2.5km weather forecast 12 collection complete
2026-10-14 08:23:51,452 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-10-14 08:23:51,784 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2026-10-13'], host='localhost')
2026-10-14 08:23:51,794 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2026-10-13'], host='localhost')
2026-10-14 08:23:51,799 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2026-10-13'], host='localhost')
2026-10-14 08:23:51,804 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2026-10-13'], host='localhost')
2026-10-14 08:23:51,811 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2026-10-14 08:23:51,816 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2026-10-14 08:23:51,822 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2026-10-14 08:23:51,829 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2026-10-14 08:23:51,836 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2026-10-14 08:23:52,640 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20261014/12/
2026-10-14 08:23:52,640 INFO [crop_gribs] 2026-10-14 12 GRIBs cropping complete
2026-10-14 08:23:52,643 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-10-14 08:23:53,124 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2026-10-14 08:23:54,377 INFO [collect_NeahBay_ssh] running in process 321149
2026-10-14 08:23:54,378 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:54,379 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2026-10-14 08:23:54,379 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:54,380 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2026-10-14
2026-10-14 08:23:54,382 INFO [download_live_ocean] running in process 321150
2026-10-14 08:23:54,383 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:54,385 INFO [download_live_ocean] publishing log messages to tcp://*:5563
2026-10-14 08:23:54,385 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:54,386 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2026-10-14
2026-10-14 08:23:54,387 INFO [collect_weather] running in process 321152
2026-10-14 08:23:54,388 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:54,389 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-10-14 08:23:54,389 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:54,405 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2026-10-14 08:23:55,109 INFO [download_live_ocean] downloaded 10199589 bytes from https://s3.kopah.uw.edu/liveocean-share/f2026.10.14/ubc.nc to /results/forcing/LiveOcean/downloaded/20261014/low_passed_UBC.nc
2026-10-14 08:23:55,110 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2026-10-14 08:23:55,321 INFO [collect_river_data] running in process 321146
2026-10-14 08:23:55,322 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:55,324 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-10-14 08:23:55,325 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:55,326 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2026-10-13
2026-10-14 08:23:55,353 INFO [collect_river_data] running in process 321145
2026-10-14 08:23:55,354 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:55,355 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-10-14 08:23:55,356 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:55,357 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2026-10-13
2026-10-14 08:23:55,358 INFO [collect_river_data] running in process 321147
2026-10-14 08:23:55,359 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:55,360 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-10-14 08:23:55,361 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:55,362 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2026-10-13
2026-10-14 08:23:55,380 INFO [collect_river_data] running in process 321143
2026-10-14 08:23:55,381 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:55,381 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-10-14 08:23:55,382 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:55,383 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2026-10-13
2026-10-14 08:23:55,469 INFO [make_turbidity_file] running in process 321148
2026-10-14 08:23:55,471 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:55,472 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2026-10-14 08:23:55,473 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:55,474 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2026-10-14
2026-10-14 08:23:55,590 INFO [grib_to_netcdf] running in process 321227
2026-10-14 08:23:55,591 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:55,592 INFO [grib_to_netcdf] publishing log messages to tcp://*:5570
2026-10-14 08:23:55,593 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:55,594 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-10-14 nowcast and forecast runs
2026-10-14 08:23:55,838 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2026-10-13&endDT=2026-10-13 "HTTP/1.1 200 "
2026-10-14 08:23:55,859 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2026-10-13&endDT=2026-10-13 "HTTP/1.1 200 "
2026-10-14 08:23:55,880 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2026-10-13&endDT=2026-10-13 "HTTP/1.1 200 "
2026-10-14 08:23:55,886 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2026-10-13&endDT=2026-10-13 "HTTP/1.1 200 "
2026-10-14 08:23:55,898 INFO [crop_gribs] running in process 321153
2026-10-14 08:23:55,900 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:55,901 INFO [crop_gribs] publishing log messages to tcp://*:5571
2026-10-14 08:23:55,902 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:55,903 INFO [crop_gribs] cropping 2026-10-14 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2026-10-14 08:23:55,914 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20261014/18/
2026-10-14 08:23:55,967 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2026-10-13 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2026-10-14 08:23:55,969 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2026-10-13 completed
2026-10-14 08:23:55,971 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-14 08:23:55,976 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2026-10-13 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2026-10-14 08:23:55,977 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2026-10-13 completed
2026-10-14 08:23:55,997 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2026-10-13 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2026-10-14 08:23:55,998 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2026-10-13 completed
2026-10-14 08:23:56,009 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2026-10-13 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2026-10-14 08:23:56,010 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2026-10-13 completed
2026-10-14 08:23:56,113 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-14 08:23:56,138 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20261014/low_passed_UBC.nc
2026-10-14 08:23:56,140 INFO [download_live_ocean] 2026-10-14 Live Ocean file for Salish Sea western boundary downloaded
2026-10-14 08:23:56,253 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-14 08:23:56,393 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-14 08:23:56,554 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2026-10-14 08:23:56,829 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:23:56,962 INFO [collect_NeahBay_ssh] 2026-10-14T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2026-10-14 08:23:56,964 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-10-14 08:23:57,105 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:23:58,378 INFO [make_turbidity_file] 2026-10-14 Fraser River turbidity file creation complete
2026-10-14 08:23:58,380 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2026-10-14 08:23:59,852 INFO [make_live_ocean_files] running in process 321292
2026-10-14 08:23:59,855 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:23:59,856 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2026-10-14 08:23:59,856 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:23:59,857 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2026-10-14 Live Ocean run
2026-10-14 08:24:00,445 INFO [make_ssh_files] running in process 321294
2026-10-14 08:24:00,447 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:00,448 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2026-10-14 08:24:00,449 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:00,450 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2026-10-14 NOAA Neah Bay 06Z observation and forecast values
2026-10-14 08:24:02,546 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m10d10.nc
2026-10-14 08:24:02,729 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m10d11.nc
2026-10-14 08:24:02,914 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m10d12.nc
2026-10-14 08:24:03,101 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m10d13.nc
2026-10-14 08:24:03,340 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d14.nc
2026-10-14 08:24:03,582 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d15.nc
2026-10-14 08:24:03,794 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d16.nc
2026-10-14 08:24:04,021 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d17.nc
2026-10-14 08:24:04,214 INFO [make_ssh_files] sea surface height boundary file for 2026-10-14 nowcast run created
2026-10-14 08:24:04,218 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-10-14 08:24:04,360 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-10-14 08:24:06,485 INFO [make_runoff_file] running in process 321338
2026-10-14 08:24:06,486 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:06,487 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2026-10-14 08:24:06,488 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:06,489 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-10-13
2026-10-14 08:24:08,115 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2026m10d14.nc
2026-10-14 08:24:09,422 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-10-13: /results/forcing/rivers/R202108Dailies_y2026m10d13.nc
2026-10-14 08:24:09,423 INFO [make_runoff_file] 2026-10-13 runoff file creation completed
2026-10-14 08:24:09,425 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-10-14 08:24:16,463 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m10d15.nc
2026-10-14 08:24:21,240 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m10d16.nc
2026-10-14 08:24:21,256 INFO [grib_to_netcdf] 2026-10-14 NEMO-atmos forcing file for nowcast+ created
2026-10-14 08:24:21,259 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-10-14 08:24:34,727 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m10d14.nc
2026-10-14 08:24:34,728 INFO [make_live_ocean_files] 2026-10-14 Live Ocean western boundary conditions files created
2026-10-14 08:24:34,730 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2026-10-14 08:24:34,871 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2026-10-14 08:24:34,877 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2026-10-14 08:24:34,884 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2026-10-14 08:24:34,888 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2026-10-14 08:24:36,060 INFO [ping_erddap] running in process 321378
2026-10-14 08:24:36,061 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:36,061 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-10-14 08:24:36,062 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:36,064 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2026-10-14 08:24:36,065 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-14 08:24:36,316 INFO [upload_forcing] running in process 321382
2026-10-14 08:24:36,317 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:36,318 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-10-14 08:24:36,319 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:36,330 INFO [upload_forcing] running in process 321380
2026-10-14 08:24:36,331 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:36,332 INFO [upload_forcing] publishing log messages to tcp://*:5565
2026-10-14 08:24:36,333 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:36,336 INFO [upload_forcing] running in process 321381
2026-10-14 08:24:36,337 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:36,338 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-10-14 08:24:36,339 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:38,089 INFO [upload_forcing] nowcast+ 2026-10-14 forcing files upload to optimum-hindcast completed
2026-10-14 08:24:38,092 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-14 08:24:39,015 INFO [upload_forcing] nowcast+ 2026-10-14 forcing files upload to arbutus.cloud-nowcast completed
2026-10-14 08:24:39,018 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-14 08:24:39,157 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:24:40,455 INFO [make_forcing_links] running in process 321407
2026-10-14 08:24:40,456 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:40,458 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-10-14 08:24:40,459 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:41,053 INFO [make_forcing_links] nowcast+ 2026-10-14 forcing file links on arbutus.cloud-nowcast created
2026-10-14 08:24:41,057 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-10-14 08:24:41,190 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-10-14'], host='arbutus.cloud-nowcast')
2026-10-14 08:24:42,390 INFO [run_NEMO] running in process 1405652
2026-10-14 08:24:42,391 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:42,392 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-10-14 08:24:42,394 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:44,540 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/14oct26nowcast_2026-10-14T082442.459778-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-10-14 08:24:44,559 INFO [run_NEMO] nowcast NEMO run for 2026-10-14 on arbutus.cloud-nowcast started
2026-10-14 08:24:44,563 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-10-14 08:24:44,631 INFO [upload_forcing] nowcast+ 2026-10-14 forcing files upload to robot.nibi completed
2026-10-14 08:24:44,708 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-10-14 08:24:44,714 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-14 08:24:45,517 INFO [watch_NEMO] running in process 1405797
2026-10-14 08:24:45,520 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:24:45,521 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-10-14 08:24:45,522 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:24:45,617 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-10-14 08:29:45,626 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19559889 = 2026-10-14 12:05:20 UTC, 50.4% complete
2026-10-14 08:34:45,628 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 19560960 = 2026-10-14 23:59:20 UTC, 100.0% complete
2026-10-14 08:39:45,634 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2026-10-14 08:39:45,639 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-10-14 08:39:46,099 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:39:46,106 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:39:47,309 INFO [download_results] running in process 321854
2026-10-14 08:39:47,310 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:39:47,312 INFO [download_results] publishing log messages to tcp://*:5562
2026-10-14 08:39:47,313 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:39:47,315 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/14oct26 to /results/SalishSea/nowcast-blue.202111
2026-10-14 08:39:47,452 INFO [make_forcing_links] running in process 321853
2026-10-14 08:39:47,454 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:39:47,456 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-10-14 08:39:47,457 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:39:48,527 INFO [make_forcing_links] ssh 2026-10-14 forcing file links on arbutus.cloud-nowcast created
2026-10-14 08:39:48,567 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-10-14 08:39:48,735 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-10-14'], host='arbutus.cloud-nowcast')
2026-10-14 08:39:49,702 INFO [run_NEMO] running in process 1406052
2026-10-14 08:39:49,715 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:39:49,730 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-10-14 08:39:49,746 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:39:51,758 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/14oct26forecast_2026-10-14T083949.793920-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-10-14 08:39:51,801 INFO [run_NEMO] forecast NEMO run for 2026-10-14 on arbutus.cloud-nowcast started
2026-10-14 08:39:51,808 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-10-14 08:39:51,958 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-10-14 08:39:52,804 INFO [watch_NEMO] running in process 1406195
2026-10-14 08:39:52,806 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:39:52,808 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-10-14 08:39:52,810 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:39:52,903 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-10-14 08:40:15,343 INFO [download_results] nowcast 2026-10-14 results files from arbutus.cloud-nowcast downloaded
2026-10-14 08:40:15,347 INFO [manager] checklist updated with [results files] items from download_results worker
2026-10-14 08:40:15,651 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:40:15,657 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2026-10-13'], host='localhost')
2026-10-14 08:40:15,663 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:40:17,658 INFO [make_CHS_currents_file] running in process 321879
2026-10-14 08:40:17,659 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:40:17,660 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-10-14 08:40:17,661 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:40:19,250 INFO [make_plots] running in process 321878
2026-10-14 08:40:19,252 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:40:19,253 INFO [make_plots] publishing log messages to tcp://*:5563
2026-10-14 08:40:19,254 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:40:19,270 INFO [make_plots] running in process 321876
2026-10-14 08:40:19,297 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:40:19,299 INFO [make_plots] publishing log messages to tcp://*:5565
2026-10-14 08:40:19,305 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:40:19,736 INFO [make_plots] preparing render list for 2026-10-13 NEMO nowcast-blue comparison figures
2026-10-14 08:40:19,737 INFO [make_plots] preparing render list for 2026-10-14 NEMO nowcast-blue research figures
2026-10-14 08:40:19,851 INFO [make_plots] added SH_wind figure to 2026-10-13 NEMO nowcast-blue comparison render list
2026-10-14 08:40:19,852 INFO [make_plots] added Compare_VENUS_East figure to 2026-10-13 NEMO nowcast-blue comparison render list
2026-10-14 08:40:19,853 INFO [make_plots] added Compare_VENUS_Central figure to 2026-10-13 NEMO nowcast-blue comparison render list
2026-10-14 08:40:19,854 INFO [make_plots] starting to render nemo nowcast comparison 13oct26 figures
2026-10-14 08:40:20,863 INFO [make_plots] added salinity_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-blue research render list
2026-10-14 08:40:21,814 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-blue research render list
2026-10-14 08:40:21,816 INFO [make_plots] added Currents_sections_and_surface figure to 2026-10-14 NEMO nowcast-blue research render list
2026-10-14 08:40:21,816 INFO [make_plots] added Currents_at_VENUS_Central figure to 2026-10-14 NEMO nowcast-blue research render list
2026-10-14 08:40:21,817 INFO [make_plots] added Currents_at_VENUS_East figure to 2026-10-14 NEMO nowcast-blue research render list
2026-10-14 08:40:21,818 INFO [make_plots] starting to render nemo nowcast research 14oct26 figures
2026-10-14 08:40:22,896 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 08:40:22,898 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 08:40:50,529 INFO [make_CHS_currents_file] Made CHS currents file for 2026-10-14 for nowcast
2026-10-14 08:40:50,531 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-10-14 08:40:51,624 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 08:40:51,626 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 08:40:51,774 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 08:40:51,776 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 08:41:23,483 INFO [make_plots] finished rendering nemo nowcast research 14oct26 figures
2026-10-14 08:41:23,485 INFO [make_plots] nemo research plots for 2026-10-14 nowcast completed
2026-10-14 08:41:23,503 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-10-14 08:41:50,101 INFO [make_plots] finished rendering nemo nowcast comparison 13oct26 figures
2026-10-14 08:41:50,104 INFO [make_plots] nemo comparison plots for 2026-10-13 nowcast completed
2026-10-14 08:41:50,108 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-10-14 08:44:52,908 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19562041 = 2026-10-15 12:00:00 UTC, 33.3% complete
2026-10-14 08:49:52,910 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 19563224 = 2026-10-16 01:08:40 UTC, 69.9% complete
2026-10-14 08:54:52,915 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2026-10-14 08:54:52,920 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-10-14 08:54:53,212 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-10-14'], host='arbutus.cloud-nowcast')
2026-10-14 08:54:53,229 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-10-14'], host='arbutus.cloud-nowcast')
2026-10-14 08:54:53,234 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:54:53,244 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:54:53,248 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:54:53,253 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:54:54,862 INFO [make_ww3_wind_file] running in process 1406529
2026-10-14 08:54:54,863 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:54:54,864 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-10-14 08:54:54,866 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:54:54,867 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-10-14 forecast run
2026-10-14 08:54:54,870 INFO [make_ww3_current_file] running in process 1406530
2026-10-14 08:54:54,871 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:54:54,872 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-10-14 08:54:54,873 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:54:54,874 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-10-14 forecast run
2026-10-14 08:54:54,906 INFO [upload_forcing] running in process 322213
2026-10-14 08:54:54,907 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:54:54,909 INFO [upload_forcing] publishing log messages to tcp://*:5562
2026-10-14 08:54:54,910 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:54:54,911 INFO [upload_forcing] running in process 322210
2026-10-14 08:54:54,912 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:54:54,913 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-10-14 08:54:54,914 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:54:54,919 INFO [download_results] running in process 322214
2026-10-14 08:54:54,920 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:54:54,921 INFO [download_results] publishing log messages to tcp://*:5565
2026-10-14 08:54:54,922 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:54:54,923 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/14oct26 to /results/SalishSea/forecast.202111
2026-10-14 08:54:54,926 INFO [upload_forcing] running in process 322212
2026-10-14 08:54:54,927 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:54:54,928 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-10-14 08:54:54,929 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:54:55,604 INFO [upload_forcing] turbidity 2026-10-14 forcing files upload to optimum-hindcast completed
2026-10-14 08:54:55,608 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-14 08:54:55,867 INFO [upload_forcing] turbidity 2026-10-14 forcing files upload to arbutus.cloud-nowcast completed
2026-10-14 08:54:55,871 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-14 08:54:56,025 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:54:57,931 INFO [make_forcing_links] running in process 322241
2026-10-14 08:54:57,933 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:54:57,934 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2026-10-14 08:54:57,936 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:54:58,170 INFO [upload_forcing] turbidity 2026-10-14 forcing files upload to robot.nibi completed
2026-10-14 08:54:58,173 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-14 08:54:58,469 INFO [make_forcing_links] nowcast-green 2026-10-14 forcing file links on arbutus.cloud-nowcast created
2026-10-14 08:54:58,473 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-10-14 08:54:58,623 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-10-14'], host='arbutus.cloud-nowcast')
2026-10-14 08:54:59,743 INFO [run_NEMO] running in process 1406988
2026-10-14 08:54:59,744 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:54:59,746 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-10-14 08:54:59,747 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:55:01,943 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-10-14 forecast run
2026-10-14 08:55:01,946 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-10-14 08:55:02,217 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/14oct26nowcast-green_2026-10-14T085459.779067-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-10-14 08:55:02,262 INFO [run_NEMO] nowcast-green NEMO run for 2026-10-14 on arbutus.cloud-nowcast started
2026-10-14 08:55:02,267 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-10-14 08:55:02,428 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2026-10-14 08:55:03,187 INFO [watch_NEMO] running in process 1407150
2026-10-14 08:55:03,190 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:55:03,192 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-10-14 08:55:03,193 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:55:03,289 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-10-14 08:55:40,813 INFO [download_results] forecast 2026-10-14 results files from arbutus.cloud-nowcast downloaded
2026-10-14 08:55:40,818 INFO [manager] checklist updated with [results files] items from download_results worker
2026-10-14 08:55:40,962 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:55:43,522 INFO [make_CHS_currents_file] running in process 322270
2026-10-14 08:55:43,523 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:55:43,524 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2026-10-14 08:55:43,525 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:56:29,065 INFO [make_CHS_currents_file] Made CHS currents file for 2026-10-14 for forecast
2026-10-14 08:56:29,069 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-10-14 08:56:29,218 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:56:30,376 INFO [update_forecast_datasets] running in process 322303
2026-10-14 08:56:30,377 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:56:30,378 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-10-14 08:56:30,379 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:56:30,380 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/14oct26 run
2026-10-14 08:56:30,869 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/14oct26 run: /results/SalishSea/rolling-forecasts/nemo
2026-10-14 08:56:30,871 INFO [update_forecast_datasets] nemo 2026-10-14 forecast rolling forecast datasets updated
2026-10-14 08:56:30,873 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-10-14 08:56:31,016 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-10-14 08:56:31,020 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:56:31,025 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 08:56:32,244 INFO [ping_erddap] running in process 322308
2026-10-14 08:56:32,246 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:56:32,247 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-10-14 08:56:32,248 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:56:32,257 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-10-14 08:56:32,258 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-14 08:56:33,737 INFO [make_surface_current_tiles] running in process 322311
2026-10-14 08:56:33,737 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:56:33,738 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2026-10-14 08:56:33,739 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:56:33,766 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-10-14 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/14oct26
2026-10-14 08:56:34,841 INFO [make_plots] running in process 322310
2026-10-14 08:56:34,844 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:56:34,846 INFO [make_plots] publishing log messages to tcp://*:5563
2026-10-14 08:56:34,846 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:56:35,334 INFO [make_plots] preparing render list for 2026-10-14 NEMO forecast publish figures
2026-10-14 08:56:35,441 INFO [make_plots] added Website_thumbnail figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,442 INFO [make_plots] added Threshold_website figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,443 INFO [make_plots] added PA_tidal_predictions figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,444 INFO [make_plots] added Boundary Bay figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,445 INFO [make_plots] added Campbell River figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,446 INFO [make_plots] added Cherry Point figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,447 INFO [make_plots] added Friday Harbor figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,448 INFO [make_plots] added Halfmoon Bay figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,448 INFO [make_plots] added Nanaimo figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,449 INFO [make_plots] added Neah Bay figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,450 INFO [make_plots] added New Westminster figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,451 INFO [make_plots] added Patricia Bay figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,452 INFO [make_plots] added Point Atkinson figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,453 INFO [make_plots] added Port Renfrew figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,454 INFO [make_plots] added Sand Heads figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,454 INFO [make_plots] added Sandy Cove figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,455 INFO [make_plots] added Squamish figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,456 INFO [make_plots] added Victoria figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,457 INFO [make_plots] added Woodwards Landing figure to 2026-10-14 NEMO forecast publish render list
2026-10-14 08:56:35,458 INFO [make_plots] starting to render nemo forecast publish 14oct26 figures
2026-10-14 08:57:26,006 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-10-14 forecast run
2026-10-14 08:57:26,010 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-10-14 08:57:26,305 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-10-14'], host='arbutus.cloud-nowcast')
2026-10-14 08:57:27,315 INFO [run_ww3] running in process 1407186
2026-10-14 08:57:27,316 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:57:27,317 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-10-14 08:57:27,318 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:57:27,322 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-10-14T085727.315214-0700
2026-10-14 08:57:27,323 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2026-10-14T085727.315214-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-10-14 08:57:27,368 INFO [run_ww3] nowcast WaveWatch3 run for 2026-10-14 on arbutus.cloud-nowcast started
2026-10-14 08:57:27,372 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-10-14 08:57:27,666 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2026-10-14 08:57:28,482 INFO [watch_ww3] running in process 1407203
2026-10-14 08:57:28,485 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 08:57:28,486 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-10-14 08:57:28,487 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-10-14 08:57:28,546 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-10-14 08:59:31,849 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 08:59:31,853 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 08:59:31,859 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 08:59:31,863 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 08:59:32,156 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 08:59:32,160 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 08:59:32,164 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 08:59:32,167 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 08:59:32,183 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 08:59:32,186 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 09:00:03,294 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19559244 = 2026-10-14 04:55:20 UTC, 20.5% complete
2026-10-14 09:02:28,550 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 426, 98.6% complete
2026-10-14 09:04:34,253 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 09:04:34,255 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 09:04:35,526 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-10-14 09:04:35,527 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-10-14 09:04:54,857 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 09:04:54,881 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 09:04:59,825 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:05:03,296 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19559728 = 2026-10-14 10:18:00 UTC, 42.9% complete
2026-10-14 09:05:57,044 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-10-14 09:05:57,044 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-10-14 09:06:16,837 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:06:53,883 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:07:13,350 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:07:28,551 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2026-10-14 09:07:28,556 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-10-14 09:07:28,889 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:07:28,896 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-10-14'], host='arbutus.cloud-nowcast')
2026-10-14 09:07:29,525 INFO [run_ww3] running in process 1407408
2026-10-14 09:07:29,527 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:07:29,527 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-10-14 09:07:29,528 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:07:29,531 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-10-14T090729.525346-0700
2026-10-14 09:07:29,532 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2026-10-14T090729.525346-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-10-14 09:07:29,576 INFO [run_ww3] forecast WaveWatch3 run for 2026-10-14 on arbutus.cloud-nowcast started
2026-10-14 09:07:29,579 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-10-14 09:07:29,862 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2026-10-14 09:07:30,626 INFO [download_wwatch3_results] running in process 324421
2026-10-14 09:07:30,627 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:07:30,628 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2026-10-14 09:07:30,629 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:07:30,679 INFO [watch_ww3] running in process 1407425
2026-10-14 09:07:30,681 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:07:30,682 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-10-14 09:07:30,683 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:07:30,747 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-10-14 09:07:33,157 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:07:41,273 INFO [download_wwatch3_results] nowcast 2026-10-14 results files from arbutus.cloud-nowcast downloaded
2026-10-14 09:07:41,276 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-10-14 09:07:52,813 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:08:11,725 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:08:30,022 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:08:48,696 WARNING [make_plots] invalid station number for Squamish station: None
2026-10-14 09:08:48,697 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-10-14 09:09:04,982 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:09:06,298 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/14oct26
2026-10-14 09:09:22,361 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:09:22,839 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-10-09 00:00:00Z to 2026-10-16 11:55:00Z
2026-10-14 09:09:38,197 INFO [make_plots] finished rendering nemo forecast publish 14oct26 figures
2026-10-14 09:09:38,200 INFO [make_plots] nemo publish plots for 2026-10-14 forecast completed
2026-10-14 09:09:38,209 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-10-14 09:09:38,366 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:09:42,228 INFO [make_feeds] running in process 324602
2026-10-14 09:09:42,230 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:09:42,231 INFO [make_feeds] publishing log messages to tcp://*:5563
2026-10-14 09:09:42,232 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:09:57,141 INFO [make_feeds] ATOM feeds for 2026-10-14 forecast run completed
2026-10-14 09:09:57,145 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-10-14 09:10:03,297 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19560221 = 2026-10-14 15:46:40 UTC, 65.8% complete
2026-10-14 09:12:30,750 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 399, 61.6% complete
2026-10-14 09:14:26,721 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/14oct26
2026-10-14 09:14:26,742 INFO [make_surface_current_tiles] finished rendering of tiles for 2026-10-14 forecast into /results/nowcast-sys/figures/surface_currents/forecast/14oct26
2026-10-14 09:14:26,743 INFO [make_surface_current_tiles] surface current tile figures for 2026-10-14 forecast completed
2026-10-14 09:14:27,204 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2026-10-14 09:15:03,298 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 19560712 = 2026-10-14 21:14:00 UTC, 88.5% complete
2026-10-14 09:17:30,750 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2026-10-14 09:17:30,755 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-10-14 09:17:31,474 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:17:32,869 INFO [download_wwatch3_results] running in process 324788
2026-10-14 09:17:32,870 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:17:32,871 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2026-10-14 09:17:32,872 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:17:44,360 INFO [download_wwatch3_results] forecast 2026-10-14 results files from arbutus.cloud-nowcast downloaded
2026-10-14 09:17:44,364 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-10-14 09:17:44,649 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:17:45,796 INFO [update_forecast_datasets] running in process 324798
2026-10-14 09:17:45,798 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:17:45,799 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2026-10-14 09:17:45,800 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:17:45,801 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/14oct26 run
2026-10-14 09:17:45,805 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/14oct26 run
2026-10-14 09:17:45,806 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/14oct26 run
2026-10-14 09:17:45,908 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/14oct26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-10-14 09:17:45,910 INFO [update_forecast_datasets] wwatch3 2026-10-14 forecast rolling forecast datasets updated
2026-10-14 09:17:45,915 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-10-14 09:17:46,208 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-10-14 09:17:47,404 INFO [ping_erddap] running in process 324804
2026-10-14 09:17:47,406 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:17:47,408 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-10-14 09:17:47,409 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:17:47,411 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-10-14 09:17:47,413 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-14 09:17:47,702 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:17:51,565 INFO [make_plots] running in process 324809
2026-10-14 09:17:51,568 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:17:51,569 INFO [make_plots] publishing log messages to tcp://*:5562
2026-10-14 09:17:51,570 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:17:51,571 INFO [make_plots] preparing render list for 2026-10-14 WaveWatch3 forecast publish figures
2026-10-14 09:17:51,572 INFO [make_plots] added Halibut Bank figure to 2026-10-14 WaveWatch3 forecast publish render list
2026-10-14 09:17:51,573 INFO [make_plots] added Sentry Shoal figure to 2026-10-14 WaveWatch3 forecast publish render list
2026-10-14 09:17:51,574 INFO [make_plots] starting to render wwatch3 forecast publish 14oct26 figures
2026-10-14 09:17:52,142 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-10-14 09:17:57,201 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 09:17:57,203 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 09:17:57,222 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 09:17:57,224 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 09:17:57,659 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-10-14 09:18:03,087 INFO [make_plots] finished rendering wwatch3 forecast publish 14oct26 figures
2026-10-14 09:18:03,088 INFO [make_plots] wwatch3 publish plots for 2026-10-14 forecast completed
2026-10-14 09:18:03,091 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-10-14 09:20:03,307 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2026-10-14 09:20:03,312 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-10-14 09:20:03,778 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:20:05,195 INFO [download_results] running in process 324881
2026-10-14 09:20:05,196 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:20:05,198 INFO [download_results] publishing log messages to tcp://*:5562
2026-10-14 09:20:05,201 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:20:05,203 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/14oct26 to /results2/SalishSea/nowcast-green.202111
2026-10-14 09:22:17,092 INFO [download_results] nowcast-green 2026-10-14 results files from arbutus.cloud-nowcast downloaded
2026-10-14 09:22:17,099 INFO [manager] checklist updated with [results files] items from download_results worker
2026-10-14 09:22:17,397 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:22:17,403 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2026-10-14 09:22:17,407 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:22:17,412 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:22:17,418 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2026-10-14'], host='localhost')
2026-10-14 09:22:19,254 INFO [ping_erddap] running in process 324934
2026-10-14 09:22:19,254 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:22:19,255 INFO [ping_erddap] publishing log messages to tcp://*:5562
2026-10-14 09:22:19,256 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:22:19,260 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2026-10-14 09:22:19,261 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-14 09:22:20,725 INFO [make_averaged_dataset] running in process 324937
2026-10-14 09:22:20,726 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:22:20,727 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2026-10-14 09:22:20,728 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:22:20,729 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Oct-2026 chemistry
2026-10-14 09:22:20,741 INFO [make_averaged_dataset] running in process 324936
2026-10-14 09:22:20,742 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:22:20,743 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2026-10-14 09:22:20,744 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:22:20,744 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Oct-2026 biology
2026-10-14 09:22:20,790 INFO [make_averaged_dataset] running in process 324935
2026-10-14 09:22:20,791 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:22:20,792 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2026-10-14 09:22:20,792 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:22:20,793 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Oct-2026 physics
2026-10-14 09:22:21,811 INFO [make_plots] running in process 324932
2026-10-14 09:22:21,812 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 09:22:21,812 INFO [make_plots] publishing log messages to tcp://*:5566
2026-10-14 09:22:21,813 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-10-14 09:22:22,314 INFO [make_plots] preparing render list for 2026-10-14 NEMO nowcast-green research figures
2026-10-14 09:22:26,539 INFO [make_plots] added temperature_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:27,377 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:28,263 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:29,112 INFO [make_plots] added silicon_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:30,017 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:30,874 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:31,739 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:32,672 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:33,586 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:34,505 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:35,405 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:36,231 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:37,489 INFO [make_plots] added temperature_salinity_timeseries figure to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:37,490 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:37,491 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:37,492 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2026-10-14 NEMO nowcast-green research render list
2026-10-14 09:22:37,493 INFO [make_plots] starting to render nemo nowcast-green research 14oct26 figures
2026-10-14 09:22:38,614 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 09:22:38,616 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 09:23:03,804 INFO [make_averaged_dataset] day-averaged dataset for 14-Oct-2026 physics created
2026-10-14 09:23:03,807 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-10-14 09:23:20,425 INFO [make_averaged_dataset] day-averaged dataset for 14-Oct-2026 chemistry created
2026-10-14 09:23:20,428 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-10-14 09:23:28,837 INFO [make_averaged_dataset] day-averaged dataset for 14-Oct-2026 biology created
2026-10-14 09:23:28,842 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2026-10-14 09:45:56,829 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-14 09:45:56,831 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-14 10:31:14,641 INFO [make_plots] finished rendering nemo nowcast-green research 14oct26 figures
2026-10-14 10:31:14,652 INFO [make_plots] nemo research plots for 2026-10-14 nowcast-green completed
2026-10-14 10:31:14,706 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-10-14 14:25:01,060 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20261014/18/
2026-10-14 14:25:01,061 INFO [collect_weather] 2026-10-14 2.5km weather forecast 18 collection complete
2026-10-14 14:25:01,065 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-10-14 14:25:01,528 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20261014/18/
2026-10-14 14:25:01,529 INFO [crop_gribs] 2026-10-14 18 GRIBs cropping complete
2026-10-14 14:25:01,612 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2026-10-14 14:25:01,624 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2026-10-14 14:25:01,631 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2026-10-14 14:25:01,641 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2026-10-15'], host='localhost')
2026-10-14 14:25:01,651 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-10-14 14:25:04,359 INFO [download_weather] running in process 331450
2026-10-14 14:25:04,362 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 14:25:04,363 INFO [download_weather] publishing log messages to tcp://*:5562
2026-10-14 14:25:04,364 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-10-14 14:25:04,365 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20261014
2026-10-14 14:25:04,366 INFO [download_weather] running in process 331452
2026-10-14 14:25:04,367 INFO [collect_weather] running in process 331453
2026-10-14 14:25:04,368 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 14:25:04,368 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 14:25:04,369 INFO [download_weather] publishing log messages to tcp://*:5563
2026-10-14 14:25:04,369 INFO [collect_weather] publishing log messages to tcp://*:5564
2026-10-14 14:25:04,370 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2026-10-14 14:25:04,370 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-10-14 14:25:04,371 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20261014
2026-10-14 14:25:05,734 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2026-10-14 14:25:06,124 INFO [crop_gribs] running in process 331454
2026-10-14 14:25:06,126 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 14:25:06,127 INFO [crop_gribs] publishing log messages to tcp://*:5565
2026-10-14 14:25:06,128 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-10-14 14:25:06,128 INFO [crop_gribs] cropping 2026-10-15 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2026-10-14 14:25:06,139 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20261015/00/
2026-10-14 14:28:52,081 INFO [download_weather] 2026-10-14 1km weather forecast 12 downloads complete
2026-10-14 14:28:52,086 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-10-14 14:28:56,389 INFO [download_weather] 2026-10-14 1km weather forecast 00 downloads complete
2026-10-14 14:28:56,393 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2026-10-14 20:25:08,476 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20261015/00/
2026-10-14 20:25:08,477 INFO [collect_weather] 2026-10-15 2.5km weather forecast 00 collection complete
2026-10-14 20:25:08,481 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-10-14 20:25:08,943 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2026-10-14 20:25:08,950 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2026-10-15'], host='localhost')
2026-10-14 20:25:09,932 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20261015/00/
2026-10-14 20:25:09,933 INFO [crop_gribs] 2026-10-15 00 GRIBs cropping complete
2026-10-14 20:25:09,936 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-10-14 20:25:11,116 INFO [collect_weather] running in process 341023
2026-10-14 20:25:11,117 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 20:25:11,118 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-10-14 20:25:11,119 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-10-14 20:25:11,138 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2026-10-14 20:25:12,582 INFO [crop_gribs] running in process 341025
2026-10-14 20:25:12,584 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-14 20:25:12,585 INFO [crop_gribs] publishing log messages to tcp://*:5563
2026-10-14 20:25:12,586 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-10-14 20:25:12,587 INFO [crop_gribs] cropping 2026-10-15 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2026-10-14 20:25:12,598 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20261015/06/
2026-10-15 02:24:44,007 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20261015/06/
2026-10-15 02:24:44,035 INFO [collect_weather] 2026-10-15 2.5km weather forecast 06 collection complete
2026-10-15 02:24:44,123 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2026-10-15 02:24:44,835 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,853 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,856 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,864 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,869 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,875 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,883 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,888 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,896 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,903 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,908 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,915 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,925 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,929 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,934 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2026-10-15 02:24:44,943 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2026-10-15 02:24:44,964 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2026-10-15 02:24:44,971 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2026-10-14'], host='localhost')
2026-10-15 02:24:44,980 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2026-10-15 02:24:44,998 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2026-10-15 02:24:45,013 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2026-10-15 02:24:45,620 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20261015/06/
2026-10-15 02:24:45,620 INFO [crop_gribs] 2026-10-15 06 GRIBs cropping complete
2026-10-15 02:24:45,627 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2026-10-15 02:24:46,682 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2026-10-15 02:24:48,552 INFO [collect_weather] running in process 349072
2026-10-15 02:24:48,553 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:48,556 INFO [collect_weather] publishing log messages to tcp://*:5562
2026-10-15 02:24:48,557 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:48,559 INFO [collect_NeahBay_ssh] running in process 349071
2026-10-15 02:24:48,560 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:48,561 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2026-10-15 02:24:48,563 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:48,565 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2026-10-15
2026-10-15 02:24:48,577 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2026-10-15 02:24:49,612 INFO [collect_NeahBay_ssh] 2026-10-15T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2026-10-15 02:24:49,619 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2026-10-15 02:24:50,054 INFO [collect_river_data] running in process 349059
2026-10-15 02:24:50,056 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,057 INFO [collect_river_data] publishing log messages to tcp://*:5564
2026-10-15 02:24:50,058 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,060 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2026-10-14
2026-10-15 02:24:50,074 INFO [collect_river_data] running in process 349057
2026-10-15 02:24:50,075 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,077 INFO [collect_river_data] publishing log messages to tcp://*:5565
2026-10-15 02:24:50,078 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,080 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2026-10-14
2026-10-15 02:24:50,086 INFO [collect_river_data] running in process 349058
2026-10-15 02:24:50,086 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,087 INFO [collect_river_data] publishing log messages to tcp://*:5566
2026-10-15 02:24:50,089 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,092 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2026-10-14
2026-10-15 02:24:50,099 INFO [collect_river_data] running in process 349061
2026-10-15 02:24:50,102 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,103 INFO [collect_river_data] publishing log messages to tcp://*:5567
2026-10-15 02:24:50,104 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,106 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2026-10-14
2026-10-15 02:24:50,107 INFO [collect_river_data] running in process 349065
2026-10-15 02:24:50,108 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,109 INFO [collect_river_data] publishing log messages to tcp://*:5568
2026-10-15 02:24:50,110 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,111 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2026-10-14
2026-10-15 02:24:50,118 INFO [collect_river_data] running in process 349052
2026-10-15 02:24:50,120 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,121 INFO [collect_river_data] publishing log messages to tcp://*:5569
2026-10-15 02:24:50,123 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,124 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2026-10-14
2026-10-15 02:24:50,126 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2026-10-15'], host='localhost')
2026-10-15 02:24:50,129 INFO [collect_river_data] running in process 349066
2026-10-15 02:24:50,130 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,131 INFO [collect_river_data] publishing log messages to tcp://*:5570
2026-10-15 02:24:50,135 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,136 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2026-10-14
2026-10-15 02:24:50,138 INFO [collect_river_data] running in process 349064
2026-10-15 02:24:50,139 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,141 INFO [collect_river_data] publishing log messages to tcp://*:5571
2026-10-15 02:24:50,144 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,146 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2026-10-14
2026-10-15 02:24:50,146 INFO [collect_river_data] running in process 349051
2026-10-15 02:24:50,150 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,153 INFO [collect_river_data] publishing log messages to tcp://*:5572
2026-10-15 02:24:50,154 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,155 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2026-10-14
2026-10-15 02:24:50,156 INFO [collect_river_data] running in process 349055
2026-10-15 02:24:50,157 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,158 INFO [collect_river_data] publishing log messages to tcp://*:5573
2026-10-15 02:24:50,159 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,160 INFO [collect_river_data] Collecting ECCC Fraser river data for 2026-10-14
2026-10-15 02:24:50,161 INFO [collect_river_data] running in process 349063
2026-10-15 02:24:50,164 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,167 INFO [collect_river_data] publishing log messages to tcp://*:5574
2026-10-15 02:24:50,168 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,171 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2026-10-14
2026-10-15 02:24:50,172 INFO [collect_river_data] running in process 349056
2026-10-15 02:24:50,173 INFO [collect_river_data] running in process 349050
2026-10-15 02:24:50,174 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,179 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,179 INFO [collect_river_data] publishing log messages to tcp://*:5575
2026-10-15 02:24:50,180 INFO [collect_river_data] publishing log messages to tcp://*:5576
2026-10-15 02:24:50,183 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,185 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,186 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2026-10-14
2026-10-15 02:24:50,190 INFO [collect_river_data] Collecting ECCC Capilano river data for 2026-10-14
2026-10-15 02:24:50,191 INFO [collect_river_data] running in process 349054
2026-10-15 02:24:50,194 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,195 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:50,196 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2026-10-15 02:24:50,197 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2026-10-15 02:24:50,198 INFO [collect_river_data] publishing log messages to tcp://*:5577
2026-10-15 02:24:50,199 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2026-10-14 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2026-10-15 02:24:50,200 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2026-10-14 completed
2026-10-15 02:24:50,201 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2026-10-14 completed
2026-10-15 02:24:50,202 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,203 INFO [collect_river_data] ECCC Fraser river data collection for 2026-10-14 completed
2026-10-15 02:24:50,215 INFO [collect_river_data] Collecting ECCC Englishman river data for 2026-10-14
2026-10-15 02:24:50,218 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2026-10-14 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2026-10-15 02:24:50,219 INFO [collect_river_data] ECCC Englishman river data collection for 2026-10-14 completed
2026-10-15 02:24:50,225 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2026-10-15 02:24:50,226 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2026-10-15 02:24:50,227 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2026-10-15 02:24:50,228 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2026-10-15 02:24:50,228 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2026-10-15 02:24:50,233 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2026-10-14 completed
2026-10-15 02:24:50,234 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2026-10-15 02:24:50,235 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2026-10-14 completed
2026-10-15 02:24:50,236 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2026-10-14 completed
2026-10-15 02:24:50,237 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2026-10-14 completed
2026-10-15 02:24:50,238 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2026-10-14 completed
2026-10-15 02:24:50,240 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2026-10-14 completed
2026-10-15 02:24:50,247 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2026-10-15 02:24:50,248 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2026-10-14 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2026-10-15 02:24:50,249 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2026-10-14 completed
2026-10-15 02:24:50,250 INFO [collect_river_data] ECCC Capilano river data collection for 2026-10-14 completed
2026-10-15 02:24:50,252 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/RobertsCreek_flow
2026-10-15 02:24:50,253 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2026-10-14 completed
2026-10-15 02:24:50,301 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2026-10-14 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2026-10-15 02:24:50,302 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2026-10-14 completed
2026-10-15 02:24:50,603 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:50,676 INFO [grib_to_netcdf] running in process 349285
2026-10-15 02:24:50,677 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,678 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2026-10-15 02:24:50,679 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,680 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2026-10-15 forecast2 run
2026-10-15 02:24:50,709 INFO [get_vfpa_hadcp] running in process 349070
2026-10-15 02:24:50,710 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,712 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2026-10-15 02:24:50,714 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,715 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-10-14
2026-10-15 02:24:50,764 INFO [crop_gribs] running in process 349073
2026-10-15 02:24:50,765 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:50,766 INFO [crop_gribs] publishing log messages to tcp://*:5579
2026-10-15 02:24:50,767 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:50,767 INFO [crop_gribs] cropping 2026-10-15 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2026-10-15 02:24:50,783 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20261015/12/
2026-10-15 02:24:51,002 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:51,343 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:51,754 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:51,907 INFO [get_onc_ctd] running in process 349067
2026-10-15 02:24:51,908 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:51,909 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2026-10-15 02:24:51,909 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:51,910 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2026-10-14
2026-10-15 02:24:51,920 INFO [get_onc_ctd] running in process 349068
2026-10-15 02:24:51,921 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:51,921 INFO [get_onc_ctd] publishing log messages to tcp://*:5567
2026-10-15 02:24:51,922 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:51,923 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2026-10-14
2026-10-15 02:24:51,931 INFO [get_onc_ferry] running in process 349069
2026-10-15 02:24:51,932 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:51,933 INFO [get_onc_ferry] publishing log messages to tcp://*:5573
2026-10-15 02:24:51,934 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:51,935 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2026-10-14
2026-10-15 02:24:52,140 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:52,518 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:52,862 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:53,209 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:53,310 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2026-10-14 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202610.nc
2026-10-15 02:24:53,310 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2026-10 netcdf file
2026-10-15 02:24:53,561 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:53,881 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:54,210 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:54,545 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:54,905 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2026-10-15 02:24:55,038 INFO [make_ssh_files] running in process 349373
2026-10-15 02:24:55,039 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:55,040 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2026-10-15 02:24:55,041 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:55,041 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2026-10-15 NOAA Neah Bay 00Z observation and forecast values
2026-10-15 02:24:55,208 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2026-10-15 02:24:55,588 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2026-10-15 02:24:56,973 INFO [ping_erddap] running in process 349426
2026-10-15 02:24:56,974 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:24:56,975 INFO [ping_erddap] publishing log messages to tcp://*:5566
2026-10-15 02:24:56,977 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:24:56,978 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2026-10-15 02:24:57,140 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-15 02:24:57,559 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d10.nc
2026-10-15 02:24:57,858 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m10d11.nc
2026-10-15 02:24:58,094 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m10d12.nc
2026-10-15 02:24:58,322 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m10d13.nc
2026-10-15 02:24:58,566 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2026m10d14.nc
2026-10-15 02:24:58,801 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d15.nc
2026-10-15 02:24:59,022 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d16.nc
2026-10-15 02:24:59,244 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d17.nc
2026-10-15 02:24:59,441 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2026m10d18.nc
2026-10-15 02:24:59,721 INFO [make_ssh_files] sea surface height boundary file for 2026-10-15 forecast2 run created
2026-10-15 02:24:59,725 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2026-10-15 02:25:00,029 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2026-10-15 02:25:02,219 INFO [make_runoff_file] running in process 349445
2026-10-15 02:25:02,220 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:02,221 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2026-10-15 02:25:02,223 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:02,226 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2026-10-14
2026-10-15 02:25:03,439 ERROR [make_runoff_file] no 2026-10-14 discharge obs for SkagitMountVernon: patching
2026-10-15 02:25:03,456 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m10d16.nc
2026-10-15 02:25:04,119 ERROR [make_runoff_file] no 2026-10-14 discharge obs for SnohomishMonroe: patching
2026-10-15 02:25:04,386 ERROR [make_runoff_file] no 2026-10-14 discharge obs for NisquallyMcKenna: patching
2026-10-15 02:25:04,529 ERROR [make_runoff_file] no 2026-10-14 discharge obs for GreenwaterGreenwater: patching
2026-10-15 02:25:06,201 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2026-10-14: /results/forcing/rivers/R202108Dailies_y2026m10d14.nc
2026-10-15 02:25:06,202 INFO [make_runoff_file] 2026-10-14 runoff file creation completed
2026-10-15 02:25:06,205 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2026-10-15 02:25:06,381 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2026m10d17.nc
2026-10-15 02:25:06,388 INFO [grib_to_netcdf] 2026-10-15 NEMO-atmos forcing file for forecast2 created
2026-10-15 02:25:06,551 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2026-10-15 02:25:06,888 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2026-10-15 02:25:06,890 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'forecast2'], host='localhost')
2026-10-15 02:25:06,894 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2026-10-15 02:25:08,465 INFO [upload_forcing] running in process 349481
2026-10-15 02:25:08,469 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:08,470 INFO [upload_forcing] publishing log messages to tcp://*:5563
2026-10-15 02:25:08,472 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:08,479 INFO [upload_forcing] running in process 349483
2026-10-15 02:25:08,480 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:08,481 INFO [upload_forcing] publishing log messages to tcp://*:5564
2026-10-15 02:25:08,485 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:08,533 INFO [upload_forcing] running in process 349484
2026-10-15 02:25:08,534 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:08,535 INFO [upload_forcing] publishing log messages to tcp://*:5566
2026-10-15 02:25:08,536 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:09,877 INFO [get_onc_ctd] 2026-10-14 ONC SCVIP CTD T&S file created
2026-10-15 02:25:09,879 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-10-15 02:25:10,140 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2026m10d14.nc
2026-10-15 02:25:10,197 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2026-10-15 02:25:10,202 INFO [upload_forcing] forecast2 2026-10-15 forcing files upload to optimum-hindcast completed
2026-10-15 02:25:10,205 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-15 02:25:10,638 INFO [upload_forcing] forecast2 2026-10-15 forcing files upload to arbutus.cloud-nowcast completed
2026-10-15 02:25:10,641 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-15 02:25:11,023 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-10-15'], host='localhost')
2026-10-15 02:25:11,269 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2026-10-14
2026-10-15 02:25:11,365 INFO [get_onc_ctd] 2026-10-14 ONC SEVIP CTD T&S file created
2026-10-15 02:25:11,367 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2026-10-15 02:25:11,519 INFO [ping_erddap] running in process 349505
2026-10-15 02:25:11,520 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:11,521 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-10-15 02:25:11,522 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:11,524 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2026-10-15 02:25:11,693 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2026-10-15 02:25:11,704 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-15 02:25:12,353 INFO [make_forcing_links] running in process 349510
2026-10-15 02:25:12,354 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:12,355 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2026-10-15 02:25:12,356 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:12,872 INFO [ping_erddap] running in process 349514
2026-10-15 02:25:12,873 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:12,874 INFO [ping_erddap] publishing log messages to tcp://*:5565
2026-10-15 02:25:12,875 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:12,877 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2026-10-15 02:25:12,880 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-15 02:25:12,986 INFO [make_forcing_links] forecast2 2026-10-15 forcing file links on arbutus.cloud-nowcast created
2026-10-15 02:25:13,215 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2026-10-15 02:25:13,529 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-10-15'], host='arbutus.cloud-nowcast')
2026-10-15 02:25:14,238 INFO [upload_forcing] forecast2 2026-10-15 forcing files upload to robot.nibi completed
2026-10-15 02:25:14,243 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2026-10-15 02:25:14,782 INFO [run_NEMO] running in process 1413916
2026-10-15 02:25:14,784 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:14,786 INFO [run_NEMO] publishing log messages to tcp://*:5556
2026-10-15 02:25:14,787 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:23,467 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/14oct26forecast2_2026-10-15T022521.465535-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2026-10-15 02:25:23,513 INFO [run_NEMO] forecast2 NEMO run for 2026-10-15 on arbutus.cloud-nowcast started
2026-10-15 02:25:23,517 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2026-10-15 02:25:23,839 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-10-15 02:25:24,696 INFO [watch_NEMO] running in process 1414062
2026-10-15 02:25:24,697 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:25:24,698 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2026-10-15 02:25:24,699 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:25:24,800 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2026-10-15 02:25:34,048 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2026-10-14
2026-10-15 02:25:55,357 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2026-10-14
2026-10-15 02:26:17,219 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2026-10-14
2026-10-15 02:26:17,449 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:17,455 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:17,457 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:17,459 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2026-10-14
2026-10-15 02:26:17,682 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:17,688 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:17,691 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:17,692 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2026-10-14
2026-10-15 02:26:17,962 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:17,967 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:17,968 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2026-10-14
2026-10-15 02:26:18,189 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:18,193 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:18,195 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2026-10-14
2026-10-15 02:26:18,448 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:18,451 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2026-10-14; substituting empty dataset
2026-10-15 02:26:18,640 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2026-10-14 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-10-15 02:26:18,657 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2026-10-14 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-10-15 02:26:18,674 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2026-10-14 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-10-15 02:26:18,691 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2026-10-14 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-10-15 02:26:18,707 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2026-10-14 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-10-15 02:26:18,724 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2026-10-14 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-10-15 02:26:18,741 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2026-10-14 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2026-10-15 02:26:43,002 INFO [get_onc_ferry] 2026-10-14 ONC TWDP ferry data file created
2026-10-15 02:26:43,005 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2026-10-15 02:26:43,461 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2026-10-15 02:26:45,404 INFO [ping_erddap] running in process 349566
2026-10-15 02:26:45,405 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:26:45,407 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-10-15 02:26:45,408 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:26:45,410 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2026-10-15 02:26:45,413 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-15 02:30:24,803 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19564201 = 2026-10-16 12:00:00 UTC, 40.0% complete
2026-10-15 02:35:24,805 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 19565373 = 2026-10-17 01:01:20 UTC, 83.4% complete
2026-10-15 02:40:24,811 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2026-10-15 02:40:24,816 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2026-10-15 02:40:25,365 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-10-15'], host='arbutus.cloud-nowcast')
2026-10-15 02:40:25,382 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-10-15'], host='arbutus.cloud-nowcast')
2026-10-15 02:40:25,386 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-10-14'], host='localhost')
2026-10-15 02:40:26,703 INFO [download_results] running in process 349852
2026-10-15 02:40:26,704 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:40:26,705 INFO [download_results] publishing log messages to tcp://*:5563
2026-10-15 02:40:26,706 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:40:26,707 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/14oct26 to /results/SalishSea/forecast2.202111
2026-10-15 02:40:26,854 INFO [make_ww3_wind_file] running in process 1414261
2026-10-15 02:40:26,858 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:40:26,863 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2026-10-15 02:40:26,869 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:40:26,875 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2026-10-15 forecast2 run
2026-10-15 02:40:26,893 INFO [make_ww3_current_file] running in process 1414262
2026-10-15 02:40:26,901 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:40:26,910 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2026-10-15 02:40:26,920 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:40:26,930 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2026-10-15 forecast2 run
2026-10-15 02:40:38,117 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2026-10-15 forecast2 run
2026-10-15 02:40:38,121 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2026-10-15 02:40:59,958 INFO [download_results] forecast2 2026-10-14 results files from arbutus.cloud-nowcast downloaded
2026-10-15 02:40:59,962 INFO [manager] checklist updated with [results files] items from download_results worker
2026-10-15 02:41:00,293 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2026-10-14'], host='localhost')
2026-10-15 02:41:03,294 INFO [make_CHS_currents_file] running in process 349868
2026-10-15 02:41:03,295 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:41:03,296 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2026-10-15 02:41:03,297 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:41:40,182 INFO [make_CHS_currents_file] Made CHS currents file for 2026-10-14 for forecast2
2026-10-15 02:41:40,185 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2026-10-15 02:41:40,514 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2026-10-14'], host='localhost')
2026-10-15 02:41:41,730 INFO [update_forecast_datasets] running in process 349902
2026-10-15 02:41:41,731 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:41:41,732 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2026-10-15 02:41:41,733 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:41:41,734 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/14oct26 run
2026-10-15 02:43:46,459 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2026-10-15 forecast2 run
2026-10-15 02:43:46,464 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2026-10-15 02:43:46,782 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-10-15'], host='arbutus.cloud-nowcast')
2026-10-15 02:43:47,650 INFO [run_ww3] running in process 1414658
2026-10-15 02:43:47,667 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:43:47,685 INFO [run_ww3] publishing log messages to tcp://*:5572
2026-10-15 02:43:47,703 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:43:47,717 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-10-15T024347.651516-0700
2026-10-15 02:43:47,719 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2026-10-15T024347.651516-0700/SoGWW3.sh on arbutus.cloud-nowcast
2026-10-15 02:43:47,721 INFO [run_ww3] forecast2 WaveWatch3 run for 2026-10-15 on arbutus.cloud-nowcast started
2026-10-15 02:43:47,722 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2026-10-15 02:43:48,039 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2026-10-15 02:43:48,897 INFO [watch_ww3] running in process 1414675
2026-10-15 02:43:48,898 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:43:48,899 INFO [watch_ww3] publishing log messages to tcp://*:5573
2026-10-15 02:43:48,901 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:43:48,955 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2026-10-15 02:46:23,788 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/14oct26 run: /results/SalishSea/rolling-forecasts/nemo
2026-10-15 02:46:23,789 INFO [update_forecast_datasets] nemo 2026-10-14 forecast2 rolling forecast datasets updated
2026-10-15 02:46:23,792 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-10-15 02:46:24,112 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2026-10-15 02:46:24,118 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2026-10-14'], host='localhost')
2026-10-15 02:46:24,124 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2026-10-14'], host='localhost')
2026-10-15 02:46:25,510 INFO [ping_erddap] running in process 350096
2026-10-15 02:46:25,512 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:46:25,512 INFO [ping_erddap] publishing log messages to tcp://*:5563
2026-10-15 02:46:25,513 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:46:25,524 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2026-10-15 02:46:25,525 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-15 02:46:27,098 INFO [make_surface_current_tiles] running in process 350099
2026-10-15 02:46:27,100 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:46:27,102 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2026-10-15 02:46:27,103 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:46:27,166 INFO [make_surface_current_tiles] starting rendering of tiles for 2026-10-14 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/14oct26
2026-10-15 02:46:28,652 INFO [make_plots] running in process 350098
2026-10-15 02:46:28,654 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:46:28,655 INFO [make_plots] publishing log messages to tcp://*:5564
2026-10-15 02:46:28,655 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:46:29,143 INFO [make_plots] preparing render list for 2026-10-14 NEMO forecast2 publish figures
2026-10-15 02:46:29,256 INFO [make_plots] added Website_thumbnail figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,256 INFO [make_plots] added Threshold_website figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,257 INFO [make_plots] added PA_tidal_predictions figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,258 INFO [make_plots] added Boundary Bay figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,258 INFO [make_plots] added Campbell River figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,259 INFO [make_plots] added Cherry Point figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,260 INFO [make_plots] added Friday Harbor figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,260 INFO [make_plots] added Halfmoon Bay figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,261 INFO [make_plots] added Nanaimo figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,262 INFO [make_plots] added Neah Bay figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,263 INFO [make_plots] added New Westminster figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,263 INFO [make_plots] added Patricia Bay figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,264 INFO [make_plots] added Point Atkinson figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,265 INFO [make_plots] added Port Renfrew figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,265 INFO [make_plots] added Sand Heads figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,266 INFO [make_plots] added Sandy Cove figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,267 INFO [make_plots] added Squamish figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,268 INFO [make_plots] added Victoria figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,268 INFO [make_plots] added Woodwards Landing figure to 2026-10-14 NEMO forecast2 publish render list
2026-10-15 02:46:29,269 INFO [make_plots] starting to render nemo forecast2 publish 14oct26 figures
2026-10-15 02:48:48,958 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 408, 75.6% complete
2026-10-15 02:49:09,831 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:49:09,833 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:49:09,837 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:49:09,839 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:49:10,027 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:49:10,029 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:49:10,031 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:49:10,033 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:49:10,042 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:49:10,044 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:53:48,959 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2026-10-15 02:53:48,964 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2026-10-15 02:53:49,729 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2026-10-15'], host='localhost')
2026-10-15 02:53:51,334 INFO [download_wwatch3_results] running in process 351519
2026-10-15 02:53:51,335 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:53:51,336 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2026-10-15 02:53:51,337 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:54:01,221 INFO [download_wwatch3_results] forecast2 2026-10-15 results files from arbutus.cloud-nowcast downloaded
2026-10-15 02:54:01,224 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2026-10-15 02:54:01,805 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2026-10-15'], host='localhost')
2026-10-15 02:54:03,256 INFO [update_forecast_datasets] running in process 351589
2026-10-15 02:54:03,258 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:54:03,258 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2026-10-15 02:54:03,418 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:54:03,420 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/15oct26 run
2026-10-15 02:54:03,425 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/15oct26 run
2026-10-15 02:54:03,426 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/15oct26 run
2026-10-15 02:54:36,503 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/15oct26 run: /results/SalishSea/rolling-forecasts/wwatch3
2026-10-15 02:54:36,504 INFO [update_forecast_datasets] wwatch3 2026-10-15 forecast2 rolling forecast datasets updated
2026-10-15 02:54:36,507 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2026-10-15 02:54:36,869 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2026-10-15 02:54:37,992 INFO [ping_erddap] running in process 351668
2026-10-15 02:54:37,993 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:54:37,994 INFO [ping_erddap] publishing log messages to tcp://*:5565
2026-10-15 02:54:37,995 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:54:37,996 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2026-10-15 02:54:37,998 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2026-10-15 02:54:38,304 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2026-10-14'], host='localhost')
2026-10-15 02:54:41,598 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:54:41,600 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:54:41,845 INFO [make_plots] running in process 351673
2026-10-15 02:54:41,846 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 02:54:41,847 INFO [make_plots] publishing log messages to tcp://*:5565
2026-10-15 02:54:41,848 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2026-10-15 02:54:41,849 INFO [make_plots] preparing render list for 2026-10-14 WaveWatch3 forecast publish figures
2026-10-15 02:54:41,850 INFO [make_plots] added Halibut Bank figure to 2026-10-14 WaveWatch3 forecast publish render list
2026-10-15 02:54:41,851 INFO [make_plots] added Sentry Shoal figure to 2026-10-14 WaveWatch3 forecast publish render list
2026-10-15 02:54:41,852 INFO [make_plots] starting to render wwatch3 forecast publish 14oct26 figures
2026-10-15 02:54:42,332 WARNING [make_plots] invalid station number for Boundary Bay station: None
2026-10-15 02:54:42,333 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2026-10-15 02:54:42,341 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2026-10-15 02:54:48,242 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:54:48,244 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:54:48,267 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:54:48,269 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:54:48,755 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2026-10-15 02:54:53,710 INFO [make_plots] finished rendering wwatch3 forecast publish 14oct26 figures
2026-10-15 02:54:53,711 INFO [make_plots] wwatch3 publish plots for 2026-10-14 forecast completed
2026-10-15 02:54:53,712 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-10-15 02:54:58,667 WARNING [make_plots] findfont: Failed to find font weight medium for Bitstream Vera Sans, now using 400.
2026-10-15 02:54:58,670 WARNING [make_plots] findfont: Failed to find font weight medium for DejaVu Sans, now using 400.
2026-10-15 02:55:02,990 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 02:56:10,427 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2026-10-15 02:56:10,428 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2026-10-15 02:56:37,738 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 02:57:26,129 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 02:57:49,403 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 02:58:14,763 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 02:58:40,034 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 02:59:00,609 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 02:59:01,498 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/14oct26
2026-10-15 02:59:17,408 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 02:59:34,619 WARNING [make_plots] invalid station number for Squamish station: None
2026-10-15 02:59:34,620 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2026-10-15 02:59:50,296 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 03:00:07,418 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 03:00:07,915 INFO [make_plots] no obs water level data available from 5cebf1e43d0f4a073c4bc42d during 2026-10-10 00:00:00Z to 2026-10-17 05:55:00Z
2026-10-15 03:00:23,693 INFO [make_plots] finished rendering nemo forecast2 publish 14oct26 figures
2026-10-15 03:00:23,696 INFO [make_plots] nemo publish plots for 2026-10-14 forecast2 completed
2026-10-15 03:00:23,706 INFO [manager] checklist updated with [plots] items from make_plots worker
2026-10-15 03:00:24,032 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2026-10-14'], host='localhost')
2026-10-15 03:00:28,524 INFO [make_feeds] running in process 352518
2026-10-15 03:00:28,526 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 03:00:28,527 INFO [make_feeds] publishing log messages to tcp://*:5564
2026-10-15 03:00:28,529 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2026-10-15 03:00:43,856 INFO [make_feeds] ATOM feeds for 2026-10-14 forecast2 run completed
2026-10-15 03:00:43,860 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2026-10-15 03:00:44,212 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2026-10-15 03:00:45,548 INFO [clear_checklist] running in process 352545
2026-10-15 03:00:45,550 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 03:00:45,551 INFO [clear_checklist] publishing log messages to tcp://*:5564
2026-10-15 03:00:45,552 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2026-10-15 03:00:45,554 INFO [clear_checklist] requesting that manager clear system state checklist
2026-10-15 03:00:45,557 INFO [manager] writing checklist to log file
2026-10-15 03:00:45,618 INFO [manager] checklist cleared
2026-10-15 03:00:45,621 INFO [clear_checklist] nowcast system checklist cleared
2026-10-15 03:00:45,624 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2026-10-15 03:00:46,963 INFO [rotate_logs] running in process 352550
2026-10-15 03:00:46,964 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2026-10-15 03:00:46,965 INFO [rotate_logs] publishing log messages to tcp://*:5564
2026-10-15 03:00:46,966 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2026-10-15 03:00:46,967 INFO [rotate_logs] rotating log files