2025-10-14 03:41:52,240 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-10-14 03:41:52,246 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-10-14 03:41:52,251 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-10-14 03:41:52,251 INFO [rotate_logs] log files rotated
2025-10-14 03:41:52,256 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-10-14 03:43:44,285 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13oct25
2025-10-14 03:43:44,308 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-10-13 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/13oct25
2025-10-14 03:43:44,309 INFO [make_surface_current_tiles] surface current tile figures for 2025-10-13 forecast2 completed
2025-10-14 03:43:44,601 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-10-14 03:46:04,425 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 408, 75.6% complete
2025-10-14 03:51:04,427 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 513, 95.0% complete
2025-10-14 03:56:04,433 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-10-14 03:56:04,436 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-10-14 03:56:04,763 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 03:56:05,869 INFO [download_wwatch3_results] running in process 3675581
2025-10-14 03:56:05,870 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 03:56:05,871 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-10-14 03:56:05,872 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-10-14 03:56:38,571 INFO [download_wwatch3_results] forecast2 2025-10-14 results files from arbutus.cloud-nowcast downloaded
2025-10-14 03:56:38,575 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-10-14 03:56:38,724 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 03:56:39,944 INFO [update_forecast_datasets] running in process 3675598
2025-10-14 03:56:39,945 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 03:56:39,947 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-10-14 03:56:39,948 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-10-14 03:56:39,949 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/14oct25 run
2025-10-14 03:56:39,954 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/14oct25 run
2025-10-14 03:56:39,955 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/14oct25 run
2025-10-14 03:57:03,495 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/14oct25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-10-14 03:57:03,495 INFO [update_forecast_datasets] wwatch3 2025-10-14 forecast2 rolling forecast datasets updated
2025-10-14 03:57:03,499 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-10-14 03:57:03,648 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-10-14 03:57:04,616 INFO [ping_erddap] running in process 3675611
2025-10-14 03:57:04,618 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 03:57:04,619 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-10-14 03:57:04,621 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-14 03:57:04,623 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-10-14 03:57:04,624 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-14 03:57:04,772 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 03:57:07,905 INFO [make_plots] running in process 3675617
2025-10-14 03:57:07,906 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 03:57:07,907 INFO [make_plots] publishing log messages to tcp://*:5562
2025-10-14 03:57:07,909 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-10-14 03:57:07,909 INFO [make_plots] preparing render list for 2025-10-14 WaveWatch3 forecast2 publish figures
2025-10-14 03:57:07,911 INFO [make_plots] added Halibut Bank figure to 2025-10-14 WaveWatch3 forecast2 publish render list
2025-10-14 03:57:07,911 INFO [make_plots] added Sentry Shoal figure to 2025-10-14 WaveWatch3 forecast2 publish render list
2025-10-14 03:57:07,912 INFO [make_plots] starting to render wwatch3 forecast2 publish 14oct25 figures
2025-10-14 03:57:08,377 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-10-14 03:57:14,068 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-10-14 03:57:18,674 INFO [make_plots] finished rendering wwatch3 forecast2 publish 14oct25 figures
2025-10-14 03:57:18,675 INFO [make_plots] wwatch3 publish plots for 2025-10-14 forecast2 completed
2025-10-14 03:57:18,679 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-10-14 08:49:42,890 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20251014/12/
2025-10-14 08:49:42,891 INFO [collect_weather] 2025-10-14 2.5km weather forecast 12 collection complete
2025-10-14 08:49:42,895 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-10-14 08:49:43,219 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-10-13'], host='localhost')
2025-10-14 08:49:43,226 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-10-13'], host='localhost')
2025-10-14 08:49:43,231 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-10-13'], host='localhost')
2025-10-14 08:49:43,234 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251014/12/
2025-10-14 08:49:43,236 INFO [crop_gribs] 2025-10-14 12 GRIBs cropping complete
2025-10-14 08:49:43,237 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-10-13'], host='localhost')
2025-10-14 08:49:43,241 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-10-14 08:49:43,246 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-10-14 08:49:43,252 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-10-14 08:49:43,257 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-10-14 08:49:43,261 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-10-14 08:49:43,270 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-10-14 08:49:43,791 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-10-14 08:49:44,921 INFO [collect_weather] running in process 3678317
2025-10-14 08:49:44,921 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:44,922 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-10-14 08:49:44,923 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:44,941 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-10-14 08:49:44,959 INFO [collect_NeahBay_ssh] running in process 3678315
2025-10-14 08:49:44,960 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:44,960 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-10-14 08:49:44,962 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:44,962 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-10-14
2025-10-14 08:49:45,109 INFO [download_live_ocean] running in process 3678316
2025-10-14 08:49:45,110 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:45,110 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-10-14 08:49:45,111 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:45,112 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-10-14
2025-10-14 08:49:45,443 INFO [collect_river_data] running in process 3678313
2025-10-14 08:49:45,444 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:45,445 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-10-14 08:49:45,446 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:45,447 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-10-13
2025-10-14 08:49:45,557 INFO [collect_river_data] running in process 3678311
2025-10-14 08:49:45,562 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:45,567 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-10-14 08:49:45,575 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:45,596 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-10-13
2025-10-14 08:49:45,615 INFO [collect_river_data] running in process 3678309
2025-10-14 08:49:45,616 INFO [collect_river_data] running in process 3678312
2025-10-14 08:49:45,617 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:45,618 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:45,619 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-10-14 08:49:45,620 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-10-14 08:49:45,621 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:45,622 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:45,623 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-10-13
2025-10-14 08:49:45,624 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-10-13
2025-10-14 08:49:45,626 INFO [download_live_ocean] downloaded 10191878 bytes from https://s3.kopah.uw.edu/liveocean-share/f2025.10.14/ubc.nc to /results/forcing/LiveOcean/downloaded/20251014/low_passed_UBC.nc
2025-10-14 08:49:45,627 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-10-14 08:49:45,664 INFO [make_turbidity_file] running in process 3678314
2025-10-14 08:49:45,664 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:45,665 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-10-14 08:49:45,666 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:45,667 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-10-14
2025-10-14 08:49:45,916 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-10-13&endDT=2025-10-13 "HTTP/1.1 200 "
2025-10-14 08:49:45,918 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-10-13&endDT=2025-10-13 "HTTP/1.1 200 "
2025-10-14 08:49:45,925 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-10-13 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-10-14 08:49:45,926 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-10-13 completed
2025-10-14 08:49:45,929 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-10-13&endDT=2025-10-13 "HTTP/1.1 200 "
2025-10-14 08:49:45,929 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-14 08:49:45,944 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-10-13 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-10-14 08:49:45,945 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-10-13 completed
2025-10-14 08:49:45,960 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-10-13 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-10-14 08:49:45,961 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-10-13 completed
2025-10-14 08:49:45,973 INFO [crop_gribs] running in process 3678318
2025-10-14 08:49:45,974 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:45,975 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-10-14 08:49:45,976 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:45,977 INFO [crop_gribs] cropping 2025-10-14 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-10-14 08:49:45,990 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251014/18/
2025-10-14 08:49:46,037 INFO [grib_to_netcdf] running in process 3678343
2025-10-14 08:49:46,038 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:46,039 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2025-10-14 08:49:46,040 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:46,041 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-10-14 nowcast and forecast runs
2025-10-14 08:49:46,064 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-14 08:49:46,072 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-10-13&endDT=2025-10-13 "HTTP/1.1 200 "
2025-10-14 08:49:46,098 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-10-13 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-10-14 08:49:46,099 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-10-13 completed
2025-10-14 08:49:46,133 INFO [collect_NeahBay_ssh] 2025-10-14T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-10-14 08:49:46,203 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-14 08:49:46,364 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-14 08:49:46,553 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-10-14 08:49:46,658 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20251014/low_passed_UBC.nc
2025-10-14 08:49:46,659 INFO [download_live_ocean] 2025-10-14 Live Ocean file for Salish Sea western boundary downloaded
2025-10-14 08:49:46,701 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 08:49:46,706 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-10-14 08:49:47,067 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-10-14'], host='localhost')
2025-10-14 08:49:48,271 INFO [make_turbidity_file] 2025-10-14 Fraser River turbidity file creation complete
2025-10-14 08:49:48,272 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-10-14 08:49:49,331 INFO [make_live_ocean_files] running in process 3678472
2025-10-14 08:49:49,332 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:49,333 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563
2025-10-14 08:49:49,334 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:49,335 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-10-14 Live Ocean run
2025-10-14 08:49:49,396 INFO [make_ssh_files] running in process 3678459
2025-10-14 08:49:49,397 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:49,398 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-10-14 08:49:49,399 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:49,399 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-10-14 NOAA Neah Bay 06Z observation and forecast values
2025-10-14 08:49:51,048 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m10d10.nc
2025-10-14 08:49:51,238 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m10d11.nc
2025-10-14 08:49:51,427 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m10d12.nc
2025-10-14 08:49:51,614 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m10d13.nc
2025-10-14 08:49:51,856 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d14.nc
2025-10-14 08:49:52,044 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d15.nc
2025-10-14 08:49:52,240 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d16.nc
2025-10-14 08:49:52,430 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d17.nc
2025-10-14 08:49:52,547 INFO [make_ssh_files] sea surface height boundary file for 2025-10-14 nowcast run created
2025-10-14 08:49:52,550 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-10-14 08:49:52,689 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-10-14 08:49:52,693 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-10-14 08:49:54,019 INFO [make_201702_runoff_file] running in process 3678538
2025-10-14 08:49:54,020 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:54,021 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-10-14 08:49:54,022 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:54,058 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-10-14 08:49:54,162 INFO [make_201702_runoff_file] 2025-10-14 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-10-14 08:49:54,165 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-10-14 08:49:54,591 INFO [make_runoff_file] running in process 3678536
2025-10-14 08:49:54,593 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:49:54,594 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-10-14 08:49:54,595 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:49:54,596 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-10-13
2025-10-14 08:49:57,272 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-10-13: /results/forcing/rivers/R202108Dailies_y2025m10d13.nc
2025-10-14 08:49:57,275 INFO [make_runoff_file] 2025-10-13 runoff file creation completed
2025-10-14 08:49:57,277 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-10-14 08:49:58,319 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m10d14.nc
2025-10-14 08:50:06,873 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m10d15.nc
2025-10-14 08:50:11,880 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m10d16.nc
2025-10-14 08:50:11,894 INFO [grib_to_netcdf] 2025-10-14 NEMO-atmos forcing file for nowcast+ created
2025-10-14 08:50:11,896 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-10-14 08:50:28,477 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m10d14.nc
2025-10-14 08:50:28,478 INFO [make_live_ocean_files] 2025-10-14 Live Ocean western boundary conditions files created
2025-10-14 08:50:28,480 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-10-14 08:50:28,633 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-10-14 08:50:28,637 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-10-14 08:50:28,642 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-10-14 08:50:28,647 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-10-14 08:50:28,651 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-10-14 08:50:29,709 INFO [ping_erddap] running in process 3678643
2025-10-14 08:50:29,710 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:50:29,711 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-10-14 08:50:29,713 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:50:29,714 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-10-14 08:50:29,716 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-14 08:50:29,850 INFO [upload_forcing] running in process 3678645
2025-10-14 08:50:29,851 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:50:29,852 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-10-14 08:50:29,853 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:50:29,860 INFO [upload_forcing] running in process 3678648
2025-10-14 08:50:29,862 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:50:29,863 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-10-14 08:50:29,864 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:50:29,872 INFO [upload_forcing] running in process 3678647
2025-10-14 08:50:29,873 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:50:29,875 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-10-14 08:50:29,876 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:50:29,901 INFO [upload_forcing] running in process 3678646
2025-10-14 08:50:29,902 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:50:29,903 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-10-14 08:50:29,903 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:50:31,583 INFO [upload_forcing] nowcast+ 2025-10-14 forcing files upload to optimum-hindcast completed
2025-10-14 08:50:31,586 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-14 08:50:35,375 INFO [upload_forcing] nowcast+ 2025-10-14 forcing files upload to arbutus.cloud-nowcast completed
2025-10-14 08:50:35,381 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-14 08:50:35,536 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 08:50:36,898 INFO [make_forcing_links] running in process 3678678
2025-10-14 08:50:36,899 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:50:36,900 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-10-14 08:50:36,902 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:50:37,245 INFO [upload_forcing] nowcast+ 2025-10-14 forcing files upload to orcinus-nowcast-agrif completed
2025-10-14 08:50:37,249 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-14 08:50:38,062 INFO [make_forcing_links] nowcast+ 2025-10-14 forcing file links on arbutus.cloud-nowcast created
2025-10-14 08:50:38,067 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-10-14 08:50:38,221 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-10-14'], host='arbutus.cloud-nowcast')
2025-10-14 08:50:40,275 INFO [run_NEMO] running in process 4093
2025-10-14 08:50:40,276 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:50:40,277 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-10-14 08:50:40,288 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:50:41,071 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-10-14 08:50:41,379 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/14oct25nowcast_2025-10-14T085040.323927-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-10-14 08:50:41,391 INFO [run_NEMO] nowcast NEMO run for 2025-10-14 on arbutus.cloud-nowcast started
2025-10-14 08:50:41,395 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-10-14 08:50:41,556 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-10-14 08:50:41,738 INFO [upload_forcing] nowcast+ 2025-10-14 forcing files upload to robot.graham completed
2025-10-14 08:50:41,742 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-14 08:50:43,509 INFO [watch_NEMO] running in process 4366
2025-10-14 08:50:43,513 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 08:50:43,515 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-10-14 08:50:43,516 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-10-14 08:50:43,581 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-10-14 08:55:43,586 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18770853 = 2025-10-14 05:01:20 UTC, 20.9% complete
2025-10-14 09:00:43,585 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18771340 = 2025-10-14 10:26:00 UTC, 43.5% complete
2025-10-14 09:05:43,586 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18771848 = 2025-10-14 16:04:40 UTC, 67.0% complete
2025-10-14 09:10:43,594 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18772366 = 2025-10-14 21:50:00 UTC, 91.0% complete
2025-10-14 09:15:43,592 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-10-14 09:15:43,595 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-10-14 09:15:43,920 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:15:43,927 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:15:45,035 INFO [download_results] running in process 3679015
2025-10-14 09:15:45,036 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:15:45,038 INFO [download_results] publishing log messages to tcp://*:5563
2025-10-14 09:15:45,039 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:15:45,040 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/14oct25 to /results/SalishSea/nowcast-blue.202111
2025-10-14 09:15:45,140 INFO [make_forcing_links] running in process 3679014
2025-10-14 09:15:45,141 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:15:45,142 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-10-14 09:15:45,143 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:15:46,100 INFO [make_forcing_links] ssh 2025-10-14 forcing file links on arbutus.cloud-nowcast created
2025-10-14 09:15:46,104 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-10-14 09:15:46,259 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-10-14'], host='arbutus.cloud-nowcast')
2025-10-14 09:15:48,336 INFO [run_NEMO] running in process 4810
2025-10-14 09:15:48,337 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:15:48,339 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-10-14 09:15:48,340 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:15:49,069 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-10-14 09:15:49,387 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/14oct25forecast_2025-10-14T091548.403732-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-10-14 09:15:49,398 INFO [run_NEMO] forecast NEMO run for 2025-10-14 on arbutus.cloud-nowcast started
2025-10-14 09:15:49,402 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-10-14 09:15:49,562 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-10-14 09:15:51,520 INFO [watch_NEMO] running in process 5083
2025-10-14 09:15:51,522 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:15:51,523 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-10-14 09:15:51,524 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:15:51,589 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-10-14 09:16:42,119 INFO [download_results] nowcast 2025-10-14 results files from arbutus.cloud-nowcast downloaded
2025-10-14 09:16:42,123 INFO [manager] checklist updated with [results files] items from download_results worker
2025-10-14 09:16:42,280 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:16:42,285 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-10-13'], host='localhost')
2025-10-14 09:16:42,291 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:16:44,113 INFO [make_CHS_currents_file] running in process 3679072
2025-10-14 09:16:44,114 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:16:44,114 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-10-14 09:16:44,116 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:16:45,172 INFO [make_plots] running in process 3679069
2025-10-14 09:16:45,173 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:16:45,175 INFO [make_plots] publishing log messages to tcp://*:5564
2025-10-14 09:16:45,176 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:16:45,177 INFO [make_plots] running in process 3679071
2025-10-14 09:16:45,178 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:16:45,179 INFO [make_plots] publishing log messages to tcp://*:5565
2025-10-14 09:16:45,180 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:16:45,539 INFO [make_plots] preparing render list for 2025-10-13 NEMO nowcast-blue comparison figures
2025-10-14 09:16:45,541 INFO [make_plots] preparing render list for 2025-10-14 NEMO nowcast-blue research figures
2025-10-14 09:16:45,574 INFO [make_plots] added SH_wind figure to 2025-10-13 NEMO nowcast-blue comparison render list
2025-10-14 09:16:45,575 INFO [make_plots] added Compare_VENUS_East figure to 2025-10-13 NEMO nowcast-blue comparison render list
2025-10-14 09:16:45,576 INFO [make_plots] added Compare_VENUS_Central figure to 2025-10-13 NEMO nowcast-blue comparison render list
2025-10-14 09:16:45,577 INFO [make_plots] starting to render nemo nowcast comparison 13oct25 figures
2025-10-14 09:16:47,886 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-blue research render list
2025-10-14 09:16:48,789 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-blue research render list
2025-10-14 09:16:48,790 INFO [make_plots] added Currents_sections_and_surface figure to 2025-10-14 NEMO nowcast-blue research render list
2025-10-14 09:16:48,791 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-10-14 NEMO nowcast-blue research render list
2025-10-14 09:16:48,792 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-10-14 NEMO nowcast-blue research render list
2025-10-14 09:16:48,793 INFO [make_plots] starting to render nemo nowcast research 14oct25 figures
2025-10-14 09:17:19,185 INFO [make_CHS_currents_file] Made CHS currents file for 2025-10-14 for nowcast
2025-10-14 09:17:19,188 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-10-14 09:17:52,847 INFO [make_plots] finished rendering nemo nowcast research 14oct25 figures
2025-10-14 09:17:52,850 INFO [make_plots] nemo research plots for 2025-10-14 nowcast completed
2025-10-14 09:17:52,867 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-10-14 09:18:10,824 INFO [make_plots] finished rendering nemo nowcast comparison 13oct25 figures
2025-10-14 09:18:10,826 INFO [make_plots] nemo comparison plots for 2025-10-13 nowcast completed
2025-10-14 09:18:10,831 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-10-14 09:20:51,590 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18773012 = 2025-10-15 05:00:40 UTC, 13.9% complete
2025-10-14 09:25:51,591 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18773503 = 2025-10-15 10:28:00 UTC, 29.1% complete
2025-10-14 09:30:51,591 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18773989 = 2025-10-15 15:52:00 UTC, 44.1% complete
2025-10-14 09:35:51,592 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18774495 = 2025-10-15 21:29:20 UTC, 59.7% complete
2025-10-14 09:40:51,593 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18774987 = 2025-10-16 02:57:20 UTC, 74.9% complete
2025-10-14 09:45:51,594 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18775455 = 2025-10-16 08:09:20 UTC, 89.3% complete
2025-10-14 09:50:51,599 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18775800 = 2025-10-16 11:59:20 UTC, 100.0% complete
2025-10-14 09:55:51,599 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-10-14 09:55:51,602 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-10-14 09:55:51,927 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-10-14'], host='arbutus.cloud-nowcast')
2025-10-14 09:55:51,938 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-10-14'], host='arbutus.cloud-nowcast')
2025-10-14 09:55:51,942 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:55:51,948 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:55:51,955 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:55:51,961 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:55:51,966 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:55:53,126 INFO [download_results] running in process 3679471
2025-10-14 09:55:53,127 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:55:53,128 INFO [download_results] publishing log messages to tcp://*:5563
2025-10-14 09:55:53,129 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:55:53,130 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/14oct25 to /results/SalishSea/forecast.202111
2025-10-14 09:55:53,177 INFO [upload_forcing] running in process 3679467
2025-10-14 09:55:53,179 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:55:53,180 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-10-14 09:55:53,181 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:55:53,197 INFO [upload_forcing] running in process 3679469
2025-10-14 09:55:53,199 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:55:53,200 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-10-14 09:55:53,201 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:55:53,204 INFO [upload_forcing] running in process 3679468
2025-10-14 09:55:53,206 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:55:53,207 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-10-14 09:55:53,208 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:55:53,209 INFO [upload_forcing] running in process 3679466
2025-10-14 09:55:53,211 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:55:53,212 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-10-14 09:55:53,213 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:55:53,866 INFO [upload_forcing] turbidity 2025-10-14 forcing files upload to optimum-hindcast completed
2025-10-14 09:55:53,870 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-14 09:55:54,275 INFO [upload_forcing] turbidity 2025-10-14 forcing files upload to arbutus.cloud-nowcast completed
2025-10-14 09:55:54,280 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-14 09:55:54,443 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:55:54,790 INFO [make_ww3_wind_file] running in process 5842
2025-10-14 09:55:54,791 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:55:54,793 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-10-14 09:55:54,803 INFO [make_ww3_current_file] running in process 5844
2025-10-14 09:55:54,805 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:55:54,806 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-10-14 09:55:54,856 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:55:54,857 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-10-14 forecast run
2025-10-14 09:55:54,869 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:55:54,870 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-10-14 forecast run
2025-10-14 09:55:55,945 INFO [make_forcing_links] running in process 3679500
2025-10-14 09:55:55,946 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:55:55,947 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2025-10-14 09:55:55,949 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:55:57,328 INFO [make_forcing_links] nowcast-green 2025-10-14 forcing file links on arbutus.cloud-nowcast created
2025-10-14 09:55:57,333 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-10-14 09:55:57,523 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-10-14'], host='arbutus.cloud-nowcast')
2025-10-14 09:55:57,621 INFO [upload_forcing] turbidity 2025-10-14 forcing files upload to robot.graham completed
2025-10-14 09:55:57,625 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-14 09:55:59,331 INFO [upload_forcing] turbidity 2025-10-14 forcing files upload to orcinus-nowcast-agrif completed
2025-10-14 09:55:59,337 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-14 09:55:59,507 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:56:00,927 INFO [make_forcing_links] running in process 3679510
2025-10-14 09:56:00,928 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:56:00,930 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-10-14 09:56:00,932 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:56:01,442 INFO [run_NEMO] running in process 6457
2025-10-14 09:56:01,444 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:56:01,445 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-10-14 09:56:01,447 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:56:03,008 INFO [make_forcing_links] nowcast-agrif 2025-10-14 forcing file links on orcinus-nowcast-agrif created
2025-10-14 09:56:03,011 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-10-14 09:56:03,169 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:56:03,618 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-10-14 09:56:04,338 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/14oct25nowcast-green_2025-10-14T095601.494894-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-10-14 09:56:04,358 INFO [run_NEMO] nowcast-green NEMO run for 2025-10-14 on arbutus.cloud-nowcast started
2025-10-14 09:56:04,363 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-10-14 09:56:04,525 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-10-14 09:56:04,561 INFO [run_NEMO_agrif] running in process 3679517
2025-10-14 09:56:04,562 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:56:04,564 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2025-10-14 09:56:04,565 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:56:06,285 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-10-14 forecast run
2025-10-14 09:56:06,294 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-10-14 09:56:06,380 CRITICAL [run_NEMO_agrif] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 109, in run_NEMO_agrif
    prev_run_namelists_info = _get_prev_run_namelists_info(
        sftp_client, host_name, run_date.shift(days=-1), config
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 155, in _get_prev_run_namelists_info
    sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 840, in get
    size = self.getfo(
        remotepath,
    ...<3 lines>...
        max_concurrent_prefetch_requests,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 795, in getfo
    file_size = self.stat(remotepath).st_size
                ~~~~~~~~~^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 493, in stat
    t, msg = self._request(CMD_STAT, path)
             ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 857, in _request
    return self._read_response(num)
           ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 909, in _read_response
    self._convert_status(msg)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
2025-10-14 09:56:06,669 INFO [watch_NEMO] running in process 6755
2025-10-14 09:56:06,670 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:56:06,671 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-10-14 09:56:06,671 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:56:06,741 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-10-14 09:57:02,444 INFO [download_results] forecast 2025-10-14 results files from arbutus.cloud-nowcast downloaded
2025-10-14 09:57:02,448 INFO [manager] checklist updated with [results files] items from download_results worker
2025-10-14 09:57:02,609 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:57:04,691 INFO [make_CHS_currents_file] running in process 3679535
2025-10-14 09:57:04,692 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:57:04,693 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-10-14 09:57:04,695 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:57:51,402 INFO [make_CHS_currents_file] Made CHS currents file for 2025-10-14 for forecast
2025-10-14 09:57:51,405 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-10-14 09:57:51,567 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:57:52,762 INFO [update_forecast_datasets] running in process 3679566
2025-10-14 09:57:52,763 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:57:52,765 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-10-14 09:57:52,766 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:57:52,767 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/14oct25 run
2025-10-14 09:57:53,133 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/14oct25 run: /results/SalishSea/rolling-forecasts/nemo
2025-10-14 09:57:53,134 INFO [update_forecast_datasets] nemo 2025-10-14 forecast rolling forecast datasets updated
2025-10-14 09:57:53,137 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-10-14 09:57:53,298 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-10-14 09:57:53,303 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:57:53,308 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 09:57:54,351 INFO [ping_erddap] running in process 3679571
2025-10-14 09:57:54,354 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:57:54,355 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-10-14 09:57:54,356 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:57:54,365 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-10-14 09:57:54,367 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-14 09:57:55,475 INFO [make_surface_current_tiles] running in process 3679574
2025-10-14 09:57:55,476 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:57:55,477 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-10-14 09:57:55,478 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:57:55,480 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-10-14 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/14oct25
2025-10-14 09:57:56,666 INFO [make_plots] running in process 3679573
2025-10-14 09:57:56,667 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:57:56,674 INFO [make_plots] publishing log messages to tcp://*:5564
2025-10-14 09:57:56,676 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:57:57,102 INFO [make_plots] preparing render list for 2025-10-14 NEMO forecast publish figures
2025-10-14 09:57:57,404 INFO [make_plots] added Website_thumbnail figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,405 INFO [make_plots] added Threshold_website figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,405 INFO [make_plots] added PA_tidal_predictions figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,406 INFO [make_plots] added Boundary Bay figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,406 INFO [make_plots] added Campbell River figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,407 INFO [make_plots] added Cherry Point figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,408 INFO [make_plots] added Friday Harbor figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,408 INFO [make_plots] added Halfmoon Bay figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,409 INFO [make_plots] added Nanaimo figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,409 INFO [make_plots] added Neah Bay figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,410 INFO [make_plots] added New Westminster figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,411 INFO [make_plots] added Patricia Bay figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,411 INFO [make_plots] added Point Atkinson figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,412 INFO [make_plots] added Port Renfrew figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,413 INFO [make_plots] added Sand Heads figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,414 INFO [make_plots] added Sandy Cove figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,414 INFO [make_plots] added Squamish figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,415 INFO [make_plots] added Victoria figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,416 INFO [make_plots] added Woodwards Landing figure to 2025-10-14 NEMO forecast publish render list
2025-10-14 09:57:57,417 INFO [make_plots] starting to render nemo forecast publish 14oct25 figures
2025-10-14 09:58:21,717 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-10-14 forecast run
2025-10-14 09:58:21,720 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-10-14 09:58:22,033 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-10-14'], host='arbutus.cloud-nowcast')
2025-10-14 09:58:24,031 INFO [run_ww3] running in process 6885
2025-10-14 09:58:24,032 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:58:24,033 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-10-14 09:58:24,034 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:58:24,037 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-10-14T095824.030173-0700
2025-10-14 09:58:24,038 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-10-14T095824.030173-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-10-14 09:58:24,050 INFO [run_ww3] nowcast WaveWatch3 run for 2025-10-14 on arbutus.cloud-nowcast started
2025-10-14 09:58:24,054 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-10-14 09:58:24,211 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-10-14 09:58:26,138 INFO [watch_ww3] running in process 6972
2025-10-14 09:58:26,139 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 09:58:26,140 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-10-14 09:58:26,141 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-10-14 09:58:26,178 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-10-14 10:01:06,742 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18770560 = 2025-10-14 01:46:00 UTC, 7.4% complete
2025-10-14 10:03:26,178 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2025-10-14 10:05:41,292 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-10-14 10:05:41,292 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-10-14 10:06:06,145 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:06:06,742 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18770739 = 2025-10-14 03:45:20 UTC, 15.7% complete
2025-10-14 10:07:10,597 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-10-14 10:07:10,598 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-10-14 10:07:32,863 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:08:05,755 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/14oct25
2025-10-14 10:08:09,543 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:08:26,181 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 201, 46.5% complete
2025-10-14 10:08:27,241 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:08:44,992 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:09:02,866 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:09:19,728 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:09:37,459 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:09:54,611 WARNING [make_plots] invalid station number for Squamish station: None
2025-10-14 10:09:54,612 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-10-14 10:10:11,335 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:10:29,401 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-10-09 00:00:00Z to 2025-10-16 11:55:00Z
2025-10-14 10:10:45,510 INFO [make_plots] finished rendering nemo forecast publish 14oct25 figures
2025-10-14 10:10:45,512 INFO [make_plots] nemo publish plots for 2025-10-14 forecast completed
2025-10-14 10:10:45,522 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-10-14 10:10:45,691 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 10:10:48,731 INFO [make_feeds] running in process 3684233
2025-10-14 10:10:48,732 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 10:10:48,733 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-10-14 10:10:48,735 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-10-14 10:11:02,429 INFO [make_feeds] ATOM feeds for 2025-10-14 forecast run completed
2025-10-14 10:11:02,432 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-10-14 10:11:06,743 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18770916 = 2025-10-14 05:43:20 UTC, 23.9% complete
2025-10-14 10:12:44,993 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/14oct25
2025-10-14 10:12:45,018 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-10-14 forecast into /results/nowcast-sys/figures/surface_currents/forecast/14oct25
2025-10-14 10:12:45,019 INFO [make_surface_current_tiles] surface current tile figures for 2025-10-14 forecast completed
2025-10-14 10:12:45,305 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-10-14 10:13:26,180 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 300, 69.4% complete
2025-10-14 10:16:06,744 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18771095 = 2025-10-14 07:42:40 UTC, 32.1% complete
2025-10-14 10:18:26,181 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 408, 94.4% complete
2025-10-14 10:21:06,746 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18771274 = 2025-10-14 09:42:00 UTC, 40.4% complete
2025-10-14 10:23:26,182 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-10-14 10:23:26,186 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-10-14 10:23:26,665 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 10:23:26,672 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-10-14'], host='arbutus.cloud-nowcast')
2025-10-14 10:23:27,895 INFO [download_wwatch3_results] running in process 3684349
2025-10-14 10:23:27,896 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 10:23:27,897 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-10-14 10:23:27,899 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-10-14 10:23:28,675 INFO [run_ww3] running in process 7306
2025-10-14 10:23:28,676 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 10:23:28,677 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-10-14 10:23:28,678 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-10-14 10:23:28,682 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-10-14T102328.673392-0700
2025-10-14 10:23:28,683 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-10-14T102328.673392-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-10-14 10:23:28,694 INFO [run_ww3] forecast WaveWatch3 run for 2025-10-14 on arbutus.cloud-nowcast started
2025-10-14 10:23:28,698 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-10-14 10:23:29,012 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-10-14 10:23:30,992 INFO [watch_ww3] running in process 7461
2025-10-14 10:23:30,993 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 10:23:30,994 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-10-14 10:23:30,996 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-10-14 10:23:31,032 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-10-14 10:23:58,384 INFO [download_wwatch3_results] nowcast 2025-10-14 results files from arbutus.cloud-nowcast downloaded
2025-10-14 10:23:58,388 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-10-14 10:26:06,747 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18771450 = 2025-10-14 11:39:20 UTC, 48.6% complete
2025-10-14 10:28:31,033 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2025-10-14 10:31:06,751 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18771628 = 2025-10-14 13:38:00 UTC, 56.8% complete
2025-10-14 10:33:31,034 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2025-10-14 10:36:06,750 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18771807 = 2025-10-14 15:37:20 UTC, 65.1% complete
2025-10-14 10:38:31,035 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 279, 43.1% complete
2025-10-14 10:41:06,751 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18771985 = 2025-10-14 17:36:00 UTC, 73.4% complete
2025-10-14 10:43:31,036 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 354, 54.6% complete
2025-10-14 10:46:06,751 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18772164 = 2025-10-14 19:35:20 UTC, 81.7% complete
2025-10-14 10:48:31,037 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 432, 66.7% complete
2025-10-14 10:51:06,751 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18772342 = 2025-10-14 21:34:00 UTC, 89.9% complete
2025-10-14 10:53:31,037 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 507, 78.2% complete
2025-10-14 10:56:06,752 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18772519 = 2025-10-14 23:32:00 UTC, 98.1% complete
2025-10-14 10:58:31,038 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 588, 90.7% complete
2025-10-14 11:01:06,753 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18772560 = 2025-10-14 23:59:20 UTC, 100.0% complete
2025-10-14 11:03:31,038 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-10-14 11:06:06,760 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-10-14 11:06:06,764 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-10-14 11:06:07,250 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 11:06:08,514 INFO [download_results] running in process 3684843
2025-10-14 11:06:08,516 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:06:08,517 INFO [download_results] publishing log messages to tcp://*:5563
2025-10-14 11:06:08,518 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:06:08,519 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/14oct25 to /results2/SalishSea/nowcast-green.202111
2025-10-14 11:08:31,040 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-10-14 11:08:31,043 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-10-14 11:08:31,518 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 11:08:32,882 INFO [download_wwatch3_results] running in process 3684894
2025-10-14 11:08:32,884 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:08:32,885 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2025-10-14 11:08:32,886 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:08:57,179 INFO [download_results] nowcast-green 2025-10-14 results files from arbutus.cloud-nowcast downloaded
2025-10-14 11:08:57,186 INFO [manager] checklist updated with [results files] items from download_results worker
2025-10-14 11:08:57,507 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 11:08:57,511 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-10-14 11:08:57,516 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 11:08:57,522 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 11:08:57,527 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 11:08:58,661 INFO [ping_erddap] running in process 3684902
2025-10-14 11:08:58,662 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:08:58,663 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-10-14 11:08:58,664 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:08:58,668 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-10-14 11:08:58,670 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-14 11:08:59,952 INFO [make_averaged_dataset] running in process 3684903
2025-10-14 11:08:59,953 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:08:59,953 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2025-10-14 11:08:59,954 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:08:59,955 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Oct-2025 biology
2025-10-14 11:08:59,970 INFO [make_averaged_dataset] running in process 3684905
2025-10-14 11:08:59,971 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:08:59,972 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-10-14 11:08:59,973 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:08:59,973 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Oct-2025 physics
2025-10-14 11:08:59,982 INFO [make_averaged_dataset] running in process 3684904
2025-10-14 11:08:59,983 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:08:59,984 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2025-10-14 11:08:59,984 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:08:59,985 INFO [make_averaged_dataset] creating day-averaged dataset for 14-Oct-2025 chemistry
2025-10-14 11:09:00,764 INFO [make_plots] running in process 3684900
2025-10-14 11:09:00,765 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:09:00,766 INFO [make_plots] publishing log messages to tcp://*:5563
2025-10-14 11:09:00,767 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:09:01,112 INFO [make_plots] preparing render list for 2025-10-14 NEMO nowcast-green research figures
2025-10-14 11:09:03,250 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:04,021 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:04,784 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:05,538 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:06,354 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:07,129 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:07,904 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:08,819 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:09,695 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:10,564 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:11,395 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:12,292 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:13,293 INFO [download_wwatch3_results] forecast 2025-10-14 results files from arbutus.cloud-nowcast downloaded
2025-10-14 11:09:13,297 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-10-14 11:09:13,634 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 11:09:13,991 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:13,992 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:13,993 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:13,994 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-10-14 NEMO nowcast-green research render list
2025-10-14 11:09:13,995 INFO [make_plots] starting to render nemo nowcast-green research 14oct25 figures
2025-10-14 11:09:14,827 INFO [update_forecast_datasets] running in process 3685001
2025-10-14 11:09:14,830 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:09:14,831 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-10-14 11:09:14,832 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:09:14,833 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/14oct25 run
2025-10-14 11:09:14,836 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/14oct25 run
2025-10-14 11:09:14,837 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/14oct25 run
2025-10-14 11:09:14,881 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/14oct25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-10-14 11:09:14,883 INFO [update_forecast_datasets] wwatch3 2025-10-14 forecast rolling forecast datasets updated
2025-10-14 11:09:14,885 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-10-14 11:09:15,198 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-10-14 11:09:16,180 INFO [ping_erddap] running in process 3685006
2025-10-14 11:09:16,182 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:09:16,183 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-10-14 11:09:16,183 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:09:16,185 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-10-14 11:09:16,186 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-14 11:09:16,513 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-10-14'], host='localhost')
2025-10-14 11:09:19,409 INFO [make_plots] running in process 3685011
2025-10-14 11:09:19,410 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 11:09:19,412 INFO [make_plots] publishing log messages to tcp://*:5564
2025-10-14 11:09:19,413 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-10-14 11:09:19,414 INFO [make_plots] preparing render list for 2025-10-14 WaveWatch3 forecast publish figures
2025-10-14 11:09:19,415 INFO [make_plots] added Halibut Bank figure to 2025-10-14 WaveWatch3 forecast publish render list
2025-10-14 11:09:19,416 INFO [make_plots] added Sentry Shoal figure to 2025-10-14 WaveWatch3 forecast publish render list
2025-10-14 11:09:19,417 INFO [make_plots] starting to render wwatch3 forecast publish 14oct25 figures
2025-10-14 11:09:19,810 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-10-14 11:09:24,958 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-10-14 11:09:29,112 INFO [make_plots] finished rendering wwatch3 forecast publish 14oct25 figures
2025-10-14 11:09:29,113 INFO [make_plots] wwatch3 publish plots for 2025-10-14 forecast completed
2025-10-14 11:09:29,117 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-10-14 12:04:45,917 INFO [make_plots] finished rendering nemo nowcast-green research 14oct25 figures
2025-10-14 12:04:46,084 INFO [make_plots] nemo research plots for 2025-10-14 nowcast-green completed
2025-10-14 12:04:46,153 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-10-14 14:48:57,610 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20251014/18/
2025-10-14 14:48:57,611 INFO [collect_weather] 2025-10-14 2.5km weather forecast 18 collection complete
2025-10-14 14:48:57,615 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-10-14 14:48:58,110 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-10-14 14:48:58,117 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-10-14 14:48:58,123 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-10-14 14:48:58,128 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-10-15'], host='localhost')
2025-10-14 14:48:58,935 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251014/18/
2025-10-14 14:48:58,935 INFO [crop_gribs] 2025-10-14 18 GRIBs cropping complete
2025-10-14 14:48:58,938 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-10-14 14:48:59,728 INFO [collect_weather] running in process 3688225
2025-10-14 14:48:59,728 INFO [download_weather] running in process 3688224
2025-10-14 14:48:59,729 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 14:48:59,730 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-10-14 14:48:59,731 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-10-14 14:48:59,731 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 14:48:59,732 INFO [download_weather] publishing log messages to tcp://*:5563
2025-10-14 14:48:59,733 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-10-14 14:48:59,733 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20251014
2025-10-14 14:48:59,735 INFO [download_weather] running in process 3688222
2025-10-14 14:48:59,736 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 14:48:59,737 INFO [download_weather] publishing log messages to tcp://*:5564
2025-10-14 14:48:59,738 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-10-14 14:48:59,738 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20251014
2025-10-14 14:48:59,783 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-10-14 14:49:00,915 INFO [crop_gribs] running in process 3688226
2025-10-14 14:49:00,916 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 14:49:00,917 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-10-14 14:49:00,918 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-10-14 14:49:00,919 INFO [crop_gribs] cropping 2025-10-15 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-10-14 14:49:00,934 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251015/00/
2025-10-14 14:52:31,091 INFO [download_weather] 2025-10-14 1km weather forecast 12 downloads complete
2025-10-14 14:52:31,094 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-10-14 14:52:40,211 INFO [download_weather] 2025-10-14 1km weather forecast 00 downloads complete
2025-10-14 14:52:40,214 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-10-14 20:56:19,546 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20251015/00/
2025-10-14 20:56:19,547 INFO [collect_weather] 2025-10-15 2.5km weather forecast 00 collection complete
2025-10-14 20:56:19,551 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-10-14 20:56:20,054 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-10-14 20:56:20,060 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-10-15'], host='localhost')
2025-10-14 20:56:20,842 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251015/00/
2025-10-14 20:56:20,843 INFO [crop_gribs] 2025-10-15 00 GRIBs cropping complete
2025-10-14 20:56:20,846 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-10-14 20:56:21,241 INFO [collect_weather] running in process 3692971
2025-10-14 20:56:21,243 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 20:56:21,244 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-10-14 20:56:21,245 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-10-14 20:56:21,262 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-10-14 20:56:22,358 INFO [crop_gribs] running in process 3692973
2025-10-14 20:56:22,359 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-14 20:56:22,360 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-10-14 20:56:22,362 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-10-14 20:56:22,363 INFO [crop_gribs] cropping 2025-10-15 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-10-14 20:56:22,376 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251015/06/
2025-10-15 02:57:01,966 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20251015/06/
2025-10-15 02:57:01,967 INFO [collect_weather] 2025-10-15 2.5km weather forecast 06 collection complete
2025-10-15 02:57:01,970 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-10-15 02:57:02,475 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,481 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,486 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,491 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,495 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,501 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,506 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,519 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,520 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,522 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,537 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,539 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,551 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,569 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,599 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-10-15 02:57:02,602 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-10-15 02:57:02,626 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-10-15 02:57:02,652 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-10-14'], host='localhost')
2025-10-15 02:57:02,676 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-10-15 02:57:02,701 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-10-15 02:57:02,714 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-10-15 02:57:04,086 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251015/06/
2025-10-15 02:57:04,087 INFO [crop_gribs] 2025-10-15 06 GRIBs cropping complete
2025-10-15 02:57:04,090 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-10-15 02:57:04,911 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-10-15 02:57:05,814 INFO [collect_NeahBay_ssh] running in process 3696350
2025-10-15 02:57:05,815 INFO [collect_weather] running in process 3696351
2025-10-15 02:57:05,816 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:05,817 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:05,818 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-10-15 02:57:05,819 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-10-15 02:57:05,820 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:05,820 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:05,821 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-10-15
2025-10-15 02:57:05,847 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-10-15 02:57:06,615 INFO [collect_river_data] running in process 3696337
2025-10-15 02:57:06,616 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,617 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-10-15 02:57:06,620 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,621 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-10-14
2025-10-15 02:57:06,637 INFO [collect_river_data] running in process 3696333
2025-10-15 02:57:06,640 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,641 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-10-15 02:57:06,642 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,643 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-10-14
2025-10-15 02:57:06,644 INFO [collect_river_data] running in process 3696336
2025-10-15 02:57:06,645 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,646 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-10-15 02:57:06,647 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,648 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-10-14
2025-10-15 02:57:06,672 INFO [collect_river_data] running in process 3696334
2025-10-15 02:57:06,673 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,674 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-10-15 02:57:06,675 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,676 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-10-14
2025-10-15 02:57:06,690 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-10-15 02:57:06,691 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-10-14 completed
2025-10-15 02:57:06,693 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-10-15 02:57:06,694 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-10-14 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-10-15 02:57:06,695 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:06,695 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-10-14 completed
2025-10-15 02:57:06,696 INFO [collect_river_data] ECCC Fraser river data collection for 2025-10-14 completed
2025-10-15 02:57:06,700 INFO [collect_river_data] running in process 3696339
2025-10-15 02:57:06,703 INFO [collect_river_data] running in process 3696335
2025-10-15 02:57:06,704 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,705 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,705 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-10-15 02:57:06,706 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-10-15 02:57:06,707 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,707 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,708 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-10-14
2025-10-15 02:57:06,709 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-10-14
2025-10-15 02:57:06,713 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-10-15 02:57:06,714 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-10-14 completed
2025-10-15 02:57:06,721 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-10-14 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-10-15 02:57:06,722 INFO [collect_river_data] ECCC Englishman river data collection for 2025-10-14 completed
2025-10-15 02:57:06,725 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-10-15 02:57:06,725 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-10-14 completed
2025-10-15 02:57:06,742 INFO [collect_river_data] running in process 3696342
2025-10-15 02:57:06,743 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,743 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-10-15 02:57:06,744 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,745 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-10-14
2025-10-15 02:57:06,749 INFO [collect_river_data] running in process 3696331
2025-10-15 02:57:06,750 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,751 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-10-15 02:57:06,752 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,753 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-10-14
2025-10-15 02:57:06,772 INFO [collect_river_data] running in process 3696340
2025-10-15 02:57:06,774 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,776 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-10-15 02:57:06,777 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,780 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-10-14
2025-10-15 02:57:06,781 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-10-15 02:57:06,782 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-10-14 completed
2025-10-15 02:57:06,785 INFO [collect_river_data] running in process 3696344
2025-10-15 02:57:06,786 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,787 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-10-15 02:57:06,788 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,790 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-10-14
2025-10-15 02:57:06,790 INFO [collect_river_data] running in process 3696345
2025-10-15 02:57:06,791 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,792 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-10-15 02:57:06,794 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,796 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-10-14
2025-10-15 02:57:06,805 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-10-14 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-10-15 02:57:06,805 INFO [collect_river_data] ECCC Capilano river data collection for 2025-10-14 completed
2025-10-15 02:57:06,808 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-10-15 02:57:06,809 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-10-14 completed
2025-10-15 02:57:06,819 INFO [collect_river_data] running in process 3696343
2025-10-15 02:57:06,820 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,821 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-10-15 02:57:06,822 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,823 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-10-14
2025-10-15 02:57:06,824 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-10-15 02:57:06,825 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-10-15 02:57:06,826 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-10-14 completed
2025-10-15 02:57:06,827 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-10-14 completed
2025-10-15 02:57:06,835 INFO [collect_river_data] running in process 3696341
2025-10-15 02:57:06,836 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,836 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-10-15 02:57:06,837 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,838 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-10-14
2025-10-15 02:57:06,856 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-10-15 02:57:06,856 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-10-14 completed
2025-10-15 02:57:06,865 INFO [collect_river_data] running in process 3696338
2025-10-15 02:57:06,866 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:06,866 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-10-15 02:57:06,867 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:06,869 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-10-14
2025-10-15 02:57:06,870 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-10-15 02:57:06,870 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-10-14 completed
2025-10-15 02:57:06,900 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-10-14 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-10-15 02:57:06,900 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-10-14 completed
2025-10-15 02:57:07,096 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:07,107 INFO [get_vfpa_hadcp] running in process 3696349
2025-10-15 02:57:07,108 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:07,109 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2025-10-15 02:57:07,110 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:07,110 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-10-14
2025-10-15 02:57:07,225 INFO [crop_gribs] running in process 3696352
2025-10-15 02:57:07,226 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:07,227 INFO [crop_gribs] publishing log messages to tcp://*:5579
2025-10-15 02:57:07,228 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:07,229 INFO [crop_gribs] cropping 2025-10-15 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-10-15 02:57:07,242 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251015/12/
2025-10-15 02:57:07,496 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:07,652 INFO [get_onc_ctd] running in process 3696347
2025-10-15 02:57:07,653 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:07,654 INFO [get_onc_ctd] publishing log messages to tcp://*:5565
2025-10-15 02:57:07,655 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:07,656 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-10-14
2025-10-15 02:57:07,665 INFO [collect_NeahBay_ssh] 2025-10-15T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-10-15 02:57:07,692 INFO [get_onc_ctd] running in process 3696346
2025-10-15 02:57:07,693 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:07,694 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2025-10-15 02:57:07,695 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:07,695 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-10-14
2025-10-15 02:57:07,735 INFO [get_onc_ferry] running in process 3696348
2025-10-15 02:57:07,736 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:07,737 INFO [get_onc_ferry] publishing log messages to tcp://*:5591
2025-10-15 02:57:07,737 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:07,738 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-10-14
2025-10-15 02:57:07,806 ERROR [get_onc_ferry] request for ONC TWDP.N1 NAV data for 2025-10-14 failed: 400 Client Error:  for url: https://data.oceannetworks.ca/api/scalardata?method=getByLocation&token=8efe44eb-0fb5-4413-bba3-ae3dbe0fbb1d&locationCode=TWDP.N1&deviceCategoryCode=NAV&sensorCategoryCodes=longitude%2Clatitude&dateFrom=2025-10-14T00:00:00.000Z&dateTo=2025-10-14T23:59:00.000Z&resampleType=avg&resamplePeriod=1
2025-10-15 02:57:07,827 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:07,850 INFO [grib_to_netcdf] running in process 3696587
2025-10-15 02:57:07,851 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:07,851 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2025-10-15 02:57:07,852 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:07,853 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-10-15 forecast2 run
2025-10-15 02:57:08,167 CRITICAL [get_onc_ferry] 2025-10-14 ONC TWDP ferry data file creation failed
2025-10-15 02:57:08,401 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:08,591 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:08,915 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:09,236 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:09,598 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:09,626 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-10-14 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202510.nc
2025-10-15 02:57:09,627 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-10 netcdf file
2025-10-15 02:57:09,904 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:10,197 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:10,502 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:10,813 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:11,132 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-10-15 02:57:11,469 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-10-15 02:57:11,782 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-10-15'], host='localhost')
2025-10-15 02:57:11,792 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-10-15 02:57:12,296 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-10-15 02:57:13,401 INFO [ping_erddap] running in process 3696855
2025-10-15 02:57:13,402 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:13,403 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-10-15 02:57:13,403 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:13,405 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-10-15 02:57:13,406 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-15 02:57:14,715 INFO [make_ssh_files] running in process 3696842
2025-10-15 02:57:14,717 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:14,718 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2025-10-15 02:57:14,719 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:14,720 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-10-15 NOAA Neah Bay 00Z observation and forecast values
2025-10-15 02:57:16,667 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d10.nc
2025-10-15 02:57:16,900 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m10d11.nc
2025-10-15 02:57:17,113 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m10d12.nc
2025-10-15 02:57:17,292 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m10d13.nc
2025-10-15 02:57:17,467 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m10d14.nc
2025-10-15 02:57:17,738 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d15.nc
2025-10-15 02:57:17,925 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d16.nc
2025-10-15 02:57:18,164 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d17.nc
2025-10-15 02:57:18,342 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m10d18.nc
2025-10-15 02:57:18,495 INFO [make_ssh_files] sea surface height boundary file for 2025-10-15 forecast2 run created
2025-10-15 02:57:18,499 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-10-15 02:57:18,851 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-10-15 02:57:18,857 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-10-15 02:57:19,402 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m10d16.nc
2025-10-15 02:57:20,157 INFO [make_201702_runoff_file] running in process 3696908
2025-10-15 02:57:20,158 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:20,159 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2025-10-15 02:57:20,160 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:20,460 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-10-15 02:57:20,569 INFO [make_201702_runoff_file] 2025-10-15 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-10-15 02:57:20,572 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-10-15 02:57:20,643 INFO [make_runoff_file] running in process 3696906
2025-10-15 02:57:20,644 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:20,645 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2025-10-15 02:57:20,646 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:20,647 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-10-14
2025-10-15 02:57:21,776 ERROR [make_runoff_file] no 2025-10-14 discharge obs for SkagitMountVernon: patching
2025-10-15 02:57:22,212 ERROR [make_runoff_file] no 2025-10-14 discharge obs for SnohomishMonroe: patching
2025-10-15 02:57:22,419 ERROR [make_runoff_file] no 2025-10-14 discharge obs for NisquallyMcKenna: patching
2025-10-15 02:57:22,433 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m10d17.nc
2025-10-15 02:57:22,440 INFO [grib_to_netcdf] 2025-10-15 NEMO-atmos forcing file for forecast2 created
2025-10-15 02:57:22,442 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-10-15 02:57:22,514 ERROR [make_runoff_file] no 2025-10-14 discharge obs for GreenwaterGreenwater: patching
2025-10-15 02:57:23,796 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-10-14: /results/forcing/rivers/R202108Dailies_y2025m10d14.nc
2025-10-15 02:57:23,797 INFO [make_runoff_file] 2025-10-14 runoff file creation completed
2025-10-15 02:57:23,800 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-10-15 02:57:24,122 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-10-15 02:57:24,127 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-10-15 02:57:24,133 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-10-15 02:57:24,138 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-10-15 02:57:25,473 INFO [upload_forcing] running in process 3696952
2025-10-15 02:57:25,474 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:25,476 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-10-15 02:57:25,477 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:25,494 INFO [upload_forcing] running in process 3696951
2025-10-15 02:57:25,495 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:25,496 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-10-15 02:57:25,496 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:25,522 INFO [upload_forcing] running in process 3696953
2025-10-15 02:57:25,523 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:25,524 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-10-15 02:57:25,525 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:25,555 INFO [upload_forcing] running in process 3696950
2025-10-15 02:57:25,556 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:25,557 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-10-15 02:57:25,558 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:26,101 INFO [get_onc_ctd] 2025-10-14 ONC SEVIP CTD T&S file created
2025-10-15 02:57:26,104 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-10-15 02:57:26,113 INFO [get_onc_ctd] 2025-10-14 ONC SCVIP CTD T&S file created
2025-10-15 02:57:26,442 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-10-15 02:57:26,448 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-10-15 02:57:26,783 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-10-15 02:57:27,053 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m10d14.nc
2025-10-15 02:57:27,128 INFO [upload_forcing] forecast2 2025-10-15 forcing files upload to optimum-hindcast completed
2025-10-15 02:57:27,131 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-15 02:57:27,636 INFO [ping_erddap] running in process 3696978
2025-10-15 02:57:27,638 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:27,639 INFO [ping_erddap] publishing log messages to tcp://*:5565
2025-10-15 02:57:27,640 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:27,641 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-10-15 02:57:27,644 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-15 02:57:27,843 INFO [ping_erddap] running in process 3696983
2025-10-15 02:57:27,844 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:27,845 INFO [ping_erddap] publishing log messages to tcp://*:5566
2025-10-15 02:57:27,847 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:27,848 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-10-15 02:57:27,952 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-15 02:57:30,377 INFO [upload_forcing] forecast2 2025-10-15 forcing files upload to arbutus.cloud-nowcast completed
2025-10-15 02:57:30,380 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-15 02:57:30,667 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-10-15'], host='localhost')
2025-10-15 02:57:31,850 INFO [make_forcing_links] running in process 3696999
2025-10-15 02:57:31,851 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:31,853 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2025-10-15 02:57:31,854 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:32,225 INFO [upload_forcing] forecast2 2025-10-15 forcing files upload to orcinus-nowcast-agrif completed
2025-10-15 02:57:32,229 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-15 02:57:33,028 INFO [make_forcing_links] forecast2 2025-10-15 forcing file links on arbutus.cloud-nowcast created
2025-10-15 02:57:33,032 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-10-15 02:57:33,352 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-10-15'], host='arbutus.cloud-nowcast')
2025-10-15 02:57:35,381 INFO [run_NEMO] running in process 18828
2025-10-15 02:57:35,384 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:35,385 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-10-15 02:57:36,411 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:37,135 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-10-15 02:57:37,450 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/14oct25forecast2_2025-10-15T025736.485697-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-10-15 02:57:37,459 INFO [run_NEMO] forecast2 NEMO run for 2025-10-15 on arbutus.cloud-nowcast started
2025-10-15 02:57:37,461 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-10-15 02:57:37,795 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-10-15 02:57:39,729 INFO [watch_NEMO] running in process 19103
2025-10-15 02:57:39,731 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 02:57:39,732 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-10-15 02:57:39,733 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-10-15 02:57:39,810 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-10-15 02:57:49,396 INFO [upload_forcing] forecast2 2025-10-15 forcing files upload to robot.graham completed
2025-10-15 02:57:49,401 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-10-15 03:02:39,811 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18775183 = 2025-10-16 05:08:00 UTC, 17.1% complete
2025-10-15 03:07:39,812 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18775681 = 2025-10-16 10:40:00 UTC, 35.6% complete
2025-10-15 03:12:39,813 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18776175 = 2025-10-16 16:09:20 UTC, 53.9% complete
2025-10-15 03:17:39,814 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18776649 = 2025-10-16 21:25:20 UTC, 71.4% complete
2025-10-15 03:22:39,818 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18777156 = 2025-10-17 03:03:20 UTC, 90.2% complete
2025-10-15 03:27:39,820 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-10-15 03:27:39,823 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-10-15 03:27:40,340 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-10-15'], host='arbutus.cloud-nowcast')
2025-10-15 03:27:40,353 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-10-15'], host='arbutus.cloud-nowcast')
2025-10-15 03:27:40,358 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-10-14'], host='localhost')
2025-10-15 03:27:41,592 INFO [download_results] running in process 3697259
2025-10-15 03:27:41,593 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:27:41,595 INFO [download_results] publishing log messages to tcp://*:5563
2025-10-15 03:27:41,596 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:27:41,597 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/14oct25 to /results/SalishSea/forecast2.202111
2025-10-15 03:27:43,009 INFO [make_ww3_wind_file] running in process 19669
2025-10-15 03:27:43,011 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:27:43,012 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-10-15 03:27:43,014 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:27:43,016 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-10-15 forecast2 run
2025-10-15 03:27:43,158 INFO [make_ww3_current_file] running in process 19671
2025-10-15 03:27:43,159 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:27:43,160 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-10-15 03:27:43,162 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:27:43,163 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-10-15 forecast2 run
2025-10-15 03:27:51,372 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-10-15 forecast2 run
2025-10-15 03:27:51,376 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-10-15 03:29:28,233 INFO [download_results] forecast2 2025-10-14 results files from arbutus.cloud-nowcast downloaded
2025-10-15 03:29:28,237 INFO [manager] checklist updated with [results files] items from download_results worker
2025-10-15 03:29:28,579 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-10-14'], host='localhost')
2025-10-15 03:29:30,665 INFO [make_CHS_currents_file] running in process 3697276
2025-10-15 03:29:30,667 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:29:30,668 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-10-15 03:29:30,669 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:30:09,696 INFO [make_CHS_currents_file] Made CHS currents file for 2025-10-14 for forecast2
2025-10-15 03:30:09,699 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-10-15 03:30:09,977 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-10-14'], host='localhost')
2025-10-15 03:30:10,969 INFO [update_forecast_datasets] running in process 3697306
2025-10-15 03:30:10,970 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:30:10,971 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-10-15 03:30:10,973 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:30:10,973 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/14oct25 run
2025-10-15 03:31:19,626 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-10-15 forecast2 run
2025-10-15 03:31:19,631 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-10-15 03:31:19,978 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-10-15'], host='arbutus.cloud-nowcast')
2025-10-15 03:31:22,015 INFO [run_ww3] running in process 20269
2025-10-15 03:31:22,017 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:31:22,018 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-10-15 03:31:22,019 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:31:22,023 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-10-15T033122.014450-0700
2025-10-15 03:31:22,024 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-10-15T033122.014450-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-10-15 03:31:22,034 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-10-15 on arbutus.cloud-nowcast started
2025-10-15 03:31:22,038 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-10-15 03:31:22,365 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-10-15 03:31:24,399 INFO [watch_ww3] running in process 20356
2025-10-15 03:31:24,400 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:31:24,401 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-10-15 03:31:24,402 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:31:24,432 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-10-15 03:34:20,193 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/14oct25 run: /results/SalishSea/rolling-forecasts/nemo
2025-10-15 03:34:20,194 INFO [update_forecast_datasets] nemo 2025-10-14 forecast2 rolling forecast datasets updated
2025-10-15 03:34:20,197 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-10-15 03:34:20,550 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-10-15 03:34:20,556 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-10-14'], host='localhost')
2025-10-15 03:34:20,562 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-10-14'], host='localhost')
2025-10-15 03:34:21,870 INFO [ping_erddap] running in process 3697399
2025-10-15 03:34:21,871 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:34:21,872 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-10-15 03:34:21,873 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:34:21,883 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-10-15 03:34:21,885 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-10-15 03:34:23,094 INFO [make_surface_current_tiles] running in process 3697402
2025-10-15 03:34:23,095 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:34:23,096 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-10-15 03:34:23,097 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:34:23,114 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-10-14 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/14oct25
2025-10-15 03:34:23,977 INFO [make_plots] running in process 3697401
2025-10-15 03:34:23,978 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:34:23,979 INFO [make_plots] publishing log messages to tcp://*:5564
2025-10-15 03:34:23,979 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:34:24,399 INFO [make_plots] preparing render list for 2025-10-14 NEMO forecast2 publish figures
2025-10-15 03:34:24,701 INFO [make_plots] added Website_thumbnail figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,702 INFO [make_plots] added Threshold_website figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,703 INFO [make_plots] added PA_tidal_predictions figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,704 INFO [make_plots] added Boundary Bay figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,714 INFO [make_plots] added Campbell River figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,719 INFO [make_plots] added Cherry Point figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,720 INFO [make_plots] added Friday Harbor figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,721 INFO [make_plots] added Halfmoon Bay figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,721 INFO [make_plots] added Nanaimo figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,722 INFO [make_plots] added Neah Bay figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,723 INFO [make_plots] added New Westminster figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,724 INFO [make_plots] added Patricia Bay figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,725 INFO [make_plots] added Point Atkinson figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,727 INFO [make_plots] added Port Renfrew figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,727 INFO [make_plots] added Sand Heads figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,729 INFO [make_plots] added Sandy Cove figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,730 INFO [make_plots] added Squamish figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,731 INFO [make_plots] added Victoria figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,733 INFO [make_plots] added Woodwards Landing figure to 2025-10-14 NEMO forecast2 publish render list
2025-10-15 03:34:24,734 INFO [make_plots] starting to render nemo forecast2 publish 14oct25 figures
2025-10-15 03:36:24,432 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 66, 12.2% complete
2025-10-15 03:41:24,435 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 147, 27.2% complete
2025-10-15 03:42:01,429 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-10-15 03:42:01,430 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-10-15 03:42:22,375 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:43:30,474 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-10-15 03:43:30,475 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-10-15 03:43:53,491 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:44:38,627 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:44:59,433 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:45:13,646 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/14oct25
2025-10-15 03:45:17,053 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:45:34,310 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:45:50,565 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:46:07,862 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:46:23,825 WARNING [make_plots] invalid station number for Squamish station: None
2025-10-15 03:46:23,826 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-10-15 03:46:24,434 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 228, 42.2% complete
2025-10-15 03:46:41,658 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:46:59,012 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-10-10 00:00:00Z to 2025-10-17 05:55:00Z
2025-10-15 03:47:14,943 INFO [make_plots] finished rendering nemo forecast2 publish 14oct25 figures
2025-10-15 03:47:14,946 INFO [make_plots] nemo publish plots for 2025-10-14 forecast2 completed
2025-10-15 03:47:14,953 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-10-15 03:47:15,295 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-10-14'], host='localhost')
2025-10-15 03:47:18,528 INFO [make_feeds] running in process 3702308
2025-10-15 03:47:18,530 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:47:18,531 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-10-15 03:47:18,532 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:47:33,927 INFO [make_feeds] ATOM feeds for 2025-10-14 forecast2 run completed
2025-10-15 03:47:33,931 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-10-15 03:47:34,257 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-10-15 03:47:35,354 INFO [clear_checklist] running in process 3702333
2025-10-15 03:47:35,355 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:47:35,356 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-10-15 03:47:35,357 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:47:35,358 INFO [clear_checklist] requesting that manager clear system state checklist
2025-10-15 03:47:35,361 INFO [manager] writing checklist to log file
2025-10-15 03:47:35,408 INFO [manager] checklist cleared
2025-10-15 03:47:35,410 INFO [clear_checklist] nowcast system checklist cleared
2025-10-15 03:47:35,413 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-10-15 03:47:36,412 INFO [rotate_logs] running in process 3702338
2025-10-15 03:47:36,413 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-10-15 03:47:36,414 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-10-15 03:47:36,415 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-10-15 03:47:36,416 INFO [rotate_logs] rotating log files