2025-07-12 03:54:57,567 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2025-07-12 03:54:57,572 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2025-07-12 03:54:57,577 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2025-07-12 03:54:57,578 INFO [rotate_logs] log files rotated 2025-07-12 03:54:57,582 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2025-07-12 03:57:22,110 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/11jul25 2025-07-12 03:57:22,133 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-07-11 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/11jul25 2025-07-12 03:57:22,134 INFO [make_surface_current_tiles] surface current tile figures for 2025-07-11 forecast2 completed 2025-07-12 03:57:22,435 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2025-07-12 03:58:58,512 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 414, 76.7% complete 2025-07-12 04:03:58,513 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 516, 95.6% complete 2025-07-12 04:08:58,516 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2025-07-12 04:08:58,520 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2025-07-12 04:08:58,892 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 04:09:00,309 INFO [download_wwatch3_results] running in process 1511332 2025-07-12 04:09:00,310 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 04:09:00,311 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563 2025-07-12 04:09:00,313 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2025-07-12 04:09:35,315 INFO [download_wwatch3_results] forecast2 2025-07-12 results files from arbutus.cloud-nowcast downloaded 2025-07-12 04:09:35,319 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2025-07-12 04:09:35,480 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 04:09:36,684 INFO [update_forecast_datasets] running in process 1511346 2025-07-12 04:09:36,685 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 04:09:36,687 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2025-07-12 04:09:36,688 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2025-07-12 04:09:36,688 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/12jul25 run 2025-07-12 04:09:36,736 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/12jul25 run 2025-07-12 04:09:36,737 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/12jul25 run 2025-07-12 04:10:06,620 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/12jul25 run: /results/SalishSea/rolling-forecasts/wwatch3 2025-07-12 04:10:06,621 INFO [update_forecast_datasets] wwatch3 2025-07-12 forecast2 rolling forecast datasets updated 2025-07-12 04:10:06,625 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2025-07-12 04:10:06,754 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2025-07-12 04:10:08,053 INFO [ping_erddap] running in process 1511357 2025-07-12 04:10:08,054 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 04:10:08,056 INFO [ping_erddap] publishing log messages to tcp://*:5563 2025-07-12 04:10:08,057 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-12 04:10:08,059 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2025-07-12 04:10:08,061 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-12 04:10:08,217 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 04:10:11,172 INFO [make_plots] running in process 1511362 2025-07-12 04:10:11,173 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 04:10:11,174 INFO [make_plots] publishing log messages to tcp://*:5563 2025-07-12 04:10:11,175 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-07-12 04:10:11,176 INFO [make_plots] preparing render list for 2025-07-12 WaveWatch3 forecast2 publish figures 2025-07-12 04:10:11,177 INFO [make_plots] added Halibut Bank figure to 2025-07-12 WaveWatch3 forecast2 publish render list 2025-07-12 04:10:11,177 INFO [make_plots] added Sentry Shoal figure to 2025-07-12 WaveWatch3 forecast2 publish render list 2025-07-12 04:10:11,178 INFO [make_plots] starting to render wwatch3 forecast2 publish 12jul25 figures 2025-07-12 04:10:11,571 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2025-07-12 04:10:18,540 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2025-07-12 04:10:22,868 INFO [make_plots] finished rendering wwatch3 forecast2 publish 12jul25 figures 2025-07-12 04:10:22,869 INFO [make_plots] wwatch3 publish plots for 2025-07-12 forecast2 completed 2025-07-12 04:10:22,873 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-07-12 09:05:46,238 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250712/12/ 2025-07-12 09:05:46,239 INFO [collect_weather] 2025-07-12 2.5km weather forecast 12 collection complete 2025-07-12 09:05:46,243 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2025-07-12 09:05:46,563 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-07-11'], host='localhost') 2025-07-12 09:05:46,569 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-07-11'], host='localhost') 2025-07-12 09:05:46,575 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-07-11'], host='localhost') 2025-07-12 09:05:46,579 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-07-11'], host='localhost') 2025-07-12 09:05:46,583 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2025-07-12 09:05:46,591 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2025-07-12 09:05:46,595 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2025-07-12 09:05:46,599 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2025-07-12 09:05:46,605 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2025-07-12 09:05:47,958 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250712/12/ 2025-07-12 09:05:47,959 INFO [crop_gribs] 2025-07-12 12 GRIBs cropping complete 2025-07-12 09:05:47,961 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2025-07-12 09:05:48,129 INFO [collect_NeahBay_ssh] running in process 1514857 2025-07-12 09:05:48,130 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:48,131 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562 2025-07-12 09:05:48,131 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:48,132 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-07-12 2025-07-12 09:05:48,166 INFO [collect_weather] running in process 1514859 2025-07-12 09:05:48,167 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:48,168 INFO [collect_weather] publishing log messages to tcp://*:5563 2025-07-12 09:05:48,169 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:48,183 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2025-07-12 09:05:48,369 INFO [download_live_ocean] running in process 1514858 2025-07-12 09:05:48,370 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:48,371 INFO [download_live_ocean] publishing log messages to tcp://*:5564 2025-07-12 09:05:48,372 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:48,373 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-07-12 2025-07-12 09:05:48,375 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2025-07-12 09:05:48,710 INFO [collect_river_data] running in process 1514851 2025-07-12 09:05:48,711 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:48,712 INFO [collect_river_data] publishing log messages to tcp://*:5565 2025-07-12 09:05:48,713 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:48,714 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-07-11 2025-07-12 09:05:48,716 INFO [collect_river_data] running in process 1514853 2025-07-12 09:05:48,717 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:48,717 INFO [collect_river_data] publishing log messages to tcp://*:5566 2025-07-12 09:05:48,718 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:48,718 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-07-11 2025-07-12 09:05:48,731 INFO [collect_river_data] running in process 1514855 2025-07-12 09:05:48,732 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:48,733 INFO [collect_river_data] publishing log messages to tcp://*:5567 2025-07-12 09:05:48,734 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:48,734 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-07-11 2025-07-12 09:05:48,795 INFO [collect_river_data] running in process 1514854 2025-07-12 09:05:48,796 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:48,797 INFO [collect_river_data] publishing log messages to tcp://*:5568 2025-07-12 09:05:48,798 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:48,799 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-07-11 2025-07-12 09:05:48,879 INFO [make_turbidity_file] running in process 1514856 2025-07-12 09:05:48,881 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:48,882 INFO [make_turbidity_file] publishing log messages to tcp://*:5569 2025-07-12 09:05:48,883 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:48,884 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-07-12 2025-07-12 09:05:49,168 INFO [crop_gribs] running in process 1514861 2025-07-12 09:05:49,169 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:49,170 INFO [crop_gribs] publishing log messages to tcp://*:5570 2025-07-12 09:05:49,171 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:49,172 INFO [crop_gribs] cropping 2025-07-12 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2025-07-12 09:05:49,179 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250712/18/ 2025-07-12 09:05:49,444 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2025-07-11&endDT=2025-07-11 "HTTP/1.1 200 " 2025-07-12 09:05:49,449 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-07-11 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2025-07-12 09:05:49,450 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-07-11 completed 2025-07-12 09:05:49,451 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2025-07-11&endDT=2025-07-11 "HTTP/1.1 200 " 2025-07-12 09:05:49,452 INFO [download_live_ocean] downloaded 10197755 bytes from https://liveocean.apl.uw.edu/output/f2025.07.12/ubc.nc to /results/forcing/LiveOcean/downloaded/20250712/low_passed_UBC.nc 2025-07-12 09:05:49,455 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2025-07-12 09:05:49,456 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-12 09:05:49,457 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2025-07-11&endDT=2025-07-11 "HTTP/1.1 200 " 2025-07-12 09:05:49,462 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2025-07-11&endDT=2025-07-11 "HTTP/1.1 200 " 2025-07-12 09:05:49,476 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-07-11 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2025-07-12 09:05:49,477 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-07-11 completed 2025-07-12 09:05:49,479 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-07-11 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2025-07-12 09:05:49,479 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-07-11 completed 2025-07-12 09:05:49,481 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-07-11 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2025-07-12 09:05:49,481 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-07-11 completed 2025-07-12 09:05:49,609 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-12 09:05:49,767 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-12 09:05:49,928 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-12 09:05:49,944 INFO [collect_NeahBay_ssh] 2025-07-12T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2025-07-12 09:05:50,190 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2025-07-12 09:05:50,334 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 09:05:50,447 INFO [grib_to_netcdf] running in process 1514952 2025-07-12 09:05:50,448 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:50,449 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565 2025-07-12 09:05:50,449 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:50,450 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-07-12 nowcast and forecast runs 2025-07-12 09:05:50,471 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250712/low_passed_UBC.nc 2025-07-12 09:05:50,475 INFO [download_live_ocean] 2025-07-12 Live Ocean file for Salish Sea western boundary downloaded 2025-07-12 09:05:50,476 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2025-07-12 09:05:50,864 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-07-12'], host='localhost') 2025-07-12 09:05:51,324 INFO [make_turbidity_file] 2025-07-12 Fraser River turbidity file creation complete 2025-07-12 09:05:51,326 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2025-07-12 09:05:52,859 INFO [make_live_ocean_files] running in process 1515004 2025-07-12 09:05:52,860 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:52,861 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562 2025-07-12 09:05:52,862 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:52,863 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-07-12 Live Ocean run 2025-07-12 09:05:52,902 INFO [make_ssh_files] running in process 1514991 2025-07-12 09:05:52,914 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:52,915 INFO [make_ssh_files] publishing log messages to tcp://*:5564 2025-07-12 09:05:52,915 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:52,916 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-07-12 NOAA Neah Bay 06Z observation and forecast values 2025-07-12 09:05:54,576 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d08.nc 2025-07-12 09:05:54,751 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d09.nc 2025-07-12 09:05:54,929 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d10.nc 2025-07-12 09:05:55,112 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d11.nc 2025-07-12 09:05:55,345 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d12.nc 2025-07-12 09:05:55,524 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d13.nc 2025-07-12 09:05:55,706 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d14.nc 2025-07-12 09:05:55,883 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d15.nc 2025-07-12 09:05:55,991 INFO [make_ssh_files] sea surface height boundary file for 2025-07-12 nowcast run created 2025-07-12 09:05:55,993 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2025-07-12 09:05:56,127 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost') 2025-07-12 09:05:56,132 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost') 2025-07-12 09:05:57,392 INFO [make_201702_runoff_file] running in process 1515039 2025-07-12 09:05:57,393 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:57,394 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564 2025-07-12 09:05:57,395 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:57,423 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry 2025-07-12 09:05:57,541 INFO [make_201702_runoff_file] 2025-07-12 runoff file creation from Fraser at Hope and climatology elsewhere complete 2025-07-12 09:05:57,544 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker 2025-07-12 09:05:57,944 INFO [make_runoff_file] running in process 1515037 2025-07-12 09:05:57,945 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:05:57,946 INFO [make_runoff_file] publishing log messages to tcp://*:5564 2025-07-12 09:05:57,947 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:05:57,950 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-07-11 2025-07-12 09:06:00,684 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-07-11: /results/forcing/rivers/R202108Dailies_y2025m07d11.nc 2025-07-12 09:06:00,685 INFO [make_runoff_file] 2025-07-11 runoff file creation completed 2025-07-12 09:06:00,688 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2025-07-12 09:06:01,316 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m07d12.nc 2025-07-12 09:06:08,608 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d13.nc 2025-07-12 09:06:12,809 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d14.nc 2025-07-12 09:06:12,823 INFO [grib_to_netcdf] 2025-07-12 NEMO-atmos forcing file for nowcast+ created 2025-07-12 09:06:12,826 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2025-07-12 09:06:35,352 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d12.nc 2025-07-12 09:06:35,353 INFO [make_live_ocean_files] 2025-07-12 Live Ocean western boundary conditions files created 2025-07-12 09:06:35,355 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2025-07-12 09:06:35,517 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2025-07-12 09:06:35,521 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2025-07-12 09:06:35,524 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2025-07-12 09:06:35,529 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2025-07-12 09:06:35,536 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2025-07-12 09:06:36,692 INFO [ping_erddap] running in process 1515088 2025-07-12 09:06:36,693 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:06:36,695 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-07-12 09:06:36,696 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:06:36,697 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2025-07-12 09:06:36,699 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-12 09:06:36,701 INFO [upload_forcing] running in process 1515093 2025-07-12 09:06:36,703 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:06:36,704 INFO [upload_forcing] publishing log messages to tcp://*:5564 2025-07-12 09:06:36,705 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:06:36,746 INFO [upload_forcing] running in process 1515090 2025-07-12 09:06:36,747 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:06:36,749 INFO [upload_forcing] publishing log messages to tcp://*:5565 2025-07-12 09:06:36,750 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:06:36,751 INFO [upload_forcing] running in process 1515091 2025-07-12 09:06:36,752 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:06:36,753 INFO [upload_forcing] publishing log messages to tcp://*:5566 2025-07-12 09:06:36,753 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:06:36,754 INFO [upload_forcing] running in process 1515092 2025-07-12 09:06:36,755 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:06:36,756 INFO [upload_forcing] publishing log messages to tcp://*:5567 2025-07-12 09:06:36,757 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:06:37,689 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 146, in _upload_ssh_files ssh_sftp.upload_file(sftp_client, host_name, localpath, remotepath, logger) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 166, in upload_file sftp_client.put(os.fspath(localpath), os.fspath(remotepath)) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 759, in put return self.putfo(fl, remotepath, file_size, callback, confirm) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 714, in putfo with self.file(remotepath, "wb") as fr: ~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 372, in open t, msg = self._request(CMD_OPEN, filename, imode, attrblock) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 857, in _request return self._read_response(num) ~~~~~~~~~~~~~~~~~~~^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 909, in _read_response self._convert_status(msg) ~~~~~~~~~~~~~~~~~~~~^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 938, in _convert_status raise IOError(errno.ENOENT, text) FileNotFoundError: [Errno 2] No such file During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 104, in upload_forcing _upload_ssh_files(sftp_client, run_type, run_date, config, host_name, host_config) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 153, in _upload_ssh_files fcst.symlink_to(localpath) ~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/pathlib/_local.py", line 789, in symlink_to os.symlink(target, self, target_is_directory) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileExistsError: [Errno 17] File exists: '/results/forcing/sshNeahBay/obs/ssh_y2025m07d11.nc' -> '/results/forcing/sshNeahBay/fcst/ssh_y2025m07d11.nc' 2025-07-12 09:06:38,799 INFO [upload_forcing] nowcast+ 2025-07-12 forcing files upload to optimum-hindcast completed 2025-07-12 09:06:38,803 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-12 09:06:43,176 INFO [upload_forcing] nowcast+ 2025-07-12 forcing files upload to arbutus.cloud-nowcast completed 2025-07-12 09:06:43,180 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-12 09:06:43,339 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 09:06:44,706 INFO [make_forcing_links] running in process 1515134 2025-07-12 09:06:44,708 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:06:44,709 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2025-07-12 09:06:44,711 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:06:45,862 INFO [make_forcing_links] nowcast+ 2025-07-12 forcing file links on arbutus.cloud-nowcast created 2025-07-12 09:06:45,866 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-07-12 09:06:46,038 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-12'], host='arbutus.cloud-nowcast') 2025-07-12 09:06:47,701 INFO [upload_forcing] nowcast+ 2025-07-12 forcing files upload to robot.graham completed 2025-07-12 09:06:47,706 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-12 09:06:48,117 INFO [run_NEMO] running in process 15011 2025-07-12 09:06:48,118 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:06:48,119 INFO [run_NEMO] publishing log messages to tcp://*:5556 2025-07-12 09:06:48,179 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:06:48,809 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets 2025-07-12 09:06:49,218 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jul25nowcast_2025-07-12T090648.163129-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2025-07-12 09:06:49,230 INFO [run_NEMO] nowcast NEMO run for 2025-07-12 on arbutus.cloud-nowcast started 2025-07-12 09:06:49,234 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2025-07-12 09:06:49,397 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2025-07-12 09:06:51,384 INFO [watch_NEMO] running in process 15284 2025-07-12 09:06:51,385 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:06:51,386 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2025-07-12 09:06:51,388 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:06:51,446 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2025-07-12 09:11:51,454 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18567821 = 2025-07-12 05:06:40 UTC, 21.3% complete 2025-07-12 09:16:51,446 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18568309 = 2025-07-12 10:32:00 UTC, 43.9% complete 2025-07-12 09:21:51,447 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18568826 = 2025-07-12 16:16:40 UTC, 67.9% complete 2025-07-12 09:26:51,447 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18569333 = 2025-07-12 21:54:40 UTC, 91.3% complete 2025-07-12 09:31:51,454 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2025-07-12 09:31:51,457 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2025-07-12 09:31:51,777 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 09:31:51,785 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 09:31:52,971 INFO [download_results] running in process 1516813 2025-07-12 09:31:52,972 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:31:52,973 INFO [download_results] publishing log messages to tcp://*:5562 2025-07-12 09:31:52,974 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:31:52,975 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/12jul25 to /results/SalishSea/nowcast-blue.202111 2025-07-12 09:31:53,004 INFO [make_forcing_links] running in process 1516812 2025-07-12 09:31:53,005 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:31:53,007 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2025-07-12 09:31:53,008 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:31:53,928 INFO [make_forcing_links] ssh 2025-07-12 forcing file links on arbutus.cloud-nowcast created 2025-07-12 09:31:53,932 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-07-12 09:31:54,097 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-12'], host='arbutus.cloud-nowcast') 2025-07-12 09:31:56,171 INFO [run_NEMO] running in process 15789 2025-07-12 09:31:56,173 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:31:56,174 INFO [run_NEMO] publishing log messages to tcp://*:5556 2025-07-12 09:31:56,175 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:31:56,992 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets 2025-07-12 09:31:57,311 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jul25forecast_2025-07-12T093156.256819-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2025-07-12 09:31:57,323 INFO [run_NEMO] forecast NEMO run for 2025-07-12 on arbutus.cloud-nowcast started 2025-07-12 09:31:57,328 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2025-07-12 09:31:57,502 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2025-07-12 09:31:59,525 INFO [watch_NEMO] running in process 16062 2025-07-12 09:31:59,526 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:31:59,528 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2025-07-12 09:31:59,529 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:31:59,600 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2025-07-12 09:32:37,493 INFO [download_results] nowcast 2025-07-12 results files from arbutus.cloud-nowcast downloaded 2025-07-12 09:32:37,498 INFO [manager] checklist updated with [results files] items from download_results worker 2025-07-12 09:32:37,658 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 09:32:37,663 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-07-11'], host='localhost') 2025-07-12 09:32:37,668 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 09:32:39,527 INFO [make_CHS_currents_file] running in process 1516838 2025-07-12 09:32:39,528 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:32:39,529 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2025-07-12 09:32:39,530 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:32:40,493 INFO [make_plots] running in process 1516837 2025-07-12 09:32:40,494 INFO [make_plots] running in process 1516835 2025-07-12 09:32:40,496 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:32:40,497 INFO [make_plots] publishing log messages to tcp://*:5565 2025-07-12 09:32:40,498 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:32:40,498 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 09:32:40,499 INFO [make_plots] publishing log messages to tcp://*:5564 2025-07-12 09:32:40,500 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-07-12 09:32:40,834 INFO [make_plots] preparing render list for 2025-07-12 NEMO nowcast-blue research figures 2025-07-12 09:32:40,835 INFO [make_plots] preparing render list for 2025-07-11 NEMO nowcast-blue comparison figures 2025-07-12 09:32:40,952 INFO [make_plots] added SH_wind figure to 2025-07-11 NEMO nowcast-blue comparison render list 2025-07-12 09:32:40,954 INFO [make_plots] added Compare_VENUS_East figure to 2025-07-11 NEMO nowcast-blue comparison render list 2025-07-12 09:32:40,955 INFO [make_plots] added Compare_VENUS_Central figure to 2025-07-11 NEMO nowcast-blue comparison render list 2025-07-12 09:32:40,956 INFO [make_plots] starting to render nemo nowcast comparison 11jul25 figures 2025-07-12 09:32:42,438 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-blue research render list 2025-07-12 09:32:43,250 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-blue research render list 2025-07-12 09:32:43,250 INFO [make_plots] added Currents_sections_and_surface figure to 2025-07-12 NEMO nowcast-blue research render list 2025-07-12 09:32:43,251 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-07-12 NEMO nowcast-blue research render list 2025-07-12 09:32:43,252 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-07-12 NEMO nowcast-blue research render list 2025-07-12 09:32:43,253 INFO [make_plots] starting to render nemo nowcast research 12jul25 figures 2025-07-12 09:33:11,322 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-12 for nowcast 2025-07-12 09:33:11,325 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2025-07-12 09:33:49,600 INFO [make_plots] finished rendering nemo nowcast research 12jul25 figures 2025-07-12 09:33:49,603 INFO [make_plots] nemo research plots for 2025-07-12 nowcast completed 2025-07-12 09:33:49,615 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-07-12 09:34:10,114 INFO [make_plots] finished rendering nemo nowcast comparison 11jul25 figures 2025-07-12 09:34:10,117 INFO [make_plots] nemo comparison plots for 2025-07-11 nowcast completed 2025-07-12 09:34:10,122 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-07-12 09:36:59,597 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18569984 = 2025-07-13 05:08:40 UTC, 14.3% complete 2025-07-12 09:41:59,599 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18570479 = 2025-07-13 10:38:40 UTC, 29.6% complete 2025-07-12 09:46:59,600 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18570978 = 2025-07-13 16:11:20 UTC, 45.0% complete 2025-07-12 09:51:59,601 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18571495 = 2025-07-13 21:56:00 UTC, 60.9% complete 2025-07-12 09:56:59,602 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18571966 = 2025-07-14 03:10:00 UTC, 75.5% complete 2025-07-12 10:01:59,604 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18572493 = 2025-07-14 09:01:20 UTC, 91.8% complete 2025-07-12 10:06:59,608 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2025-07-12 10:06:59,611 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2025-07-12 10:06:59,936 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-12'], host='arbutus.cloud-nowcast') 2025-07-12 10:06:59,944 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-12'], host='arbutus.cloud-nowcast') 2025-07-12 10:06:59,949 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:06:59,956 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:06:59,962 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:06:59,967 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:06:59,972 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:07:01,211 INFO [upload_forcing] running in process 1517334 2025-07-12 10:07:01,212 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:01,213 INFO [upload_forcing] publishing log messages to tcp://*:5562 2025-07-12 10:07:01,214 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:01,215 INFO [download_results] running in process 1517335 2025-07-12 10:07:01,216 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:01,217 INFO [download_results] publishing log messages to tcp://*:5564 2025-07-12 10:07:01,218 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:01,220 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/12jul25 to /results/SalishSea/forecast.202111 2025-07-12 10:07:01,256 INFO [upload_forcing] running in process 1517332 2025-07-12 10:07:01,258 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:01,259 INFO [upload_forcing] publishing log messages to tcp://*:5565 2025-07-12 10:07:01,260 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:01,260 INFO [upload_forcing] running in process 1517333 2025-07-12 10:07:01,261 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:01,262 INFO [upload_forcing] publishing log messages to tcp://*:5566 2025-07-12 10:07:01,263 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:01,273 INFO [upload_forcing] running in process 1517330 2025-07-12 10:07:01,275 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:01,276 INFO [upload_forcing] publishing log messages to tcp://*:5567 2025-07-12 10:07:01,277 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:01,866 INFO [upload_forcing] turbidity 2025-07-12 forcing files upload to optimum-hindcast completed 2025-07-12 10:07:01,869 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-12 10:07:02,175 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 146, in _upload_ssh_files ssh_sftp.upload_file(sftp_client, host_name, localpath, remotepath, logger) ~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 166, in upload_file sftp_client.put(os.fspath(localpath), os.fspath(remotepath)) ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 759, in put return self.putfo(fl, remotepath, file_size, callback, confirm) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 714, in putfo with self.file(remotepath, "wb") as fr: ~~~~~~~~~^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 372, in open t, msg = self._request(CMD_OPEN, filename, imode, attrblock) ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 857, in _request return self._read_response(num) ~~~~~~~~~~~~~~~~~~~^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 909, in _read_response self._convert_status(msg) ~~~~~~~~~~~~~~~~~~~~^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 938, in _convert_status raise IOError(errno.ENOENT, text) FileNotFoundError: [Errno 2] No such file During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work checklist = self.worker_func( self._parsed_args, self.config, self.tell_manager ) File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 104, in upload_forcing _upload_ssh_files(sftp_client, run_type, run_date, config, host_name, host_config) ~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 153, in _upload_ssh_files fcst.symlink_to(localpath) ~~~~~~~~~~~~~~~^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.13/pathlib/_local.py", line 789, in symlink_to os.symlink(target, self, target_is_directory) ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ FileExistsError: [Errno 17] File exists: '/results/forcing/sshNeahBay/obs/ssh_y2025m07d11.nc' -> '/results/forcing/sshNeahBay/fcst/ssh_y2025m07d11.nc' 2025-07-12 10:07:02,432 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-12 10:07:02,437 INFO [upload_forcing] turbidity 2025-07-12 forcing files upload to arbutus.cloud-nowcast completed 2025-07-12 10:07:02,473 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:07:03,432 INFO [make_ww3_wind_file] running in process 16764 2025-07-12 10:07:03,434 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:03,435 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2025-07-12 10:07:03,493 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:03,494 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-07-12 forecast run 2025-07-12 10:07:03,579 INFO [make_ww3_current_file] running in process 16762 2025-07-12 10:07:03,580 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:03,581 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2025-07-12 10:07:03,582 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:03,583 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-07-12 forecast run 2025-07-12 10:07:03,850 INFO [make_forcing_links] running in process 1517371 2025-07-12 10:07:03,851 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:03,853 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2025-07-12 10:07:03,854 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:05,143 INFO [make_forcing_links] nowcast-green 2025-07-12 forcing file links on arbutus.cloud-nowcast created 2025-07-12 10:07:05,148 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-07-12 10:07:05,318 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-12'], host='arbutus.cloud-nowcast') 2025-07-12 10:07:05,771 INFO [upload_forcing] turbidity 2025-07-12 forcing files upload to robot.graham completed 2025-07-12 10:07:05,775 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-12 10:07:07,426 INFO [run_NEMO] running in process 17082 2025-07-12 10:07:07,428 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:07,429 INFO [run_NEMO] publishing log messages to tcp://*:5556 2025-07-12 10:07:07,431 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:10,064 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets 2025-07-12 10:07:10,648 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jul25nowcast-green_2025-07-12T100707.659029-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2025-07-12 10:07:10,696 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2025-07-12 10:07:10,700 INFO [run_NEMO] nowcast-green NEMO run for 2025-07-12 on arbutus.cloud-nowcast started 2025-07-12 10:07:10,863 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2025-07-12 10:07:14,313 INFO [watch_NEMO] running in process 17681 2025-07-12 10:07:14,314 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:07:14,316 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2025-07-12 10:07:14,317 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:07:14,391 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2025-07-12 10:07:16,049 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-07-12 forecast run 2025-07-12 10:07:16,053 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2025-07-12 10:09:19,588 INFO [download_results] forecast 2025-07-12 results files from arbutus.cloud-nowcast downloaded 2025-07-12 10:09:19,591 INFO [manager] checklist updated with [results files] items from download_results worker 2025-07-12 10:09:19,760 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:09:22,072 INFO [make_CHS_currents_file] running in process 1517420 2025-07-12 10:09:22,073 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:09:22,074 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2025-07-12 10:09:22,075 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:09:37,536 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-07-12 forecast run 2025-07-12 10:09:37,538 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2025-07-12 10:09:37,708 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-12'], host='arbutus.cloud-nowcast') 2025-07-12 10:09:39,585 INFO [run_ww3] running in process 17807 2025-07-12 10:09:39,587 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:09:39,588 INFO [run_ww3] publishing log messages to tcp://*:5572 2025-07-12 10:09:39,589 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:09:39,616 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-07-12T100939.581726-0700 2025-07-12 10:09:39,617 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-07-12T100939.581726-0700/SoGWW3.sh on arbutus.cloud-nowcast 2025-07-12 10:09:39,626 INFO [run_ww3] nowcast WaveWatch3 run for 2025-07-12 on arbutus.cloud-nowcast started 2025-07-12 10:09:39,635 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2025-07-12 10:09:39,810 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2025-07-12 10:09:41,675 INFO [watch_ww3] running in process 17894 2025-07-12 10:09:41,677 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:09:41,678 INFO [watch_ww3] publishing log messages to tcp://*:5573 2025-07-12 10:09:41,679 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:09:41,721 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2025-07-12 10:10:03,867 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-12 for forecast 2025-07-12 10:10:03,870 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2025-07-12 10:10:04,042 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:10:05,068 INFO [update_forecast_datasets] running in process 1517444 2025-07-12 10:10:05,070 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:10:05,071 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2025-07-12 10:10:05,072 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:10:05,073 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/12jul25 run 2025-07-12 10:10:05,093 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/12jul25 run: /results/SalishSea/rolling-forecasts/nemo 2025-07-12 10:10:05,094 INFO [update_forecast_datasets] nemo 2025-07-12 forecast rolling forecast datasets updated 2025-07-12 10:10:05,096 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2025-07-12 10:10:05,226 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2025-07-12 10:10:05,232 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:10:05,237 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:10:06,345 INFO [ping_erddap] running in process 1517449 2025-07-12 10:10:06,346 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:10:06,347 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-07-12 10:10:06,348 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:10:06,357 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2025-07-12 10:10:06,359 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-12 10:10:07,473 INFO [make_surface_current_tiles] running in process 1517452 2025-07-12 10:10:07,474 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:10:07,475 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562 2025-07-12 10:10:07,475 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:10:07,478 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-07-12 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/12jul25 2025-07-12 10:10:08,206 INFO [make_plots] running in process 1517451 2025-07-12 10:10:08,207 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:10:08,208 INFO [make_plots] publishing log messages to tcp://*:5564 2025-07-12 10:10:08,214 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:10:08,667 INFO [make_plots] preparing render list for 2025-07-12 NEMO forecast publish figures 2025-07-12 10:10:08,798 INFO [make_plots] added Website_thumbnail figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,805 INFO [make_plots] added Threshold_website figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,807 INFO [make_plots] added PA_tidal_predictions figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,808 INFO [make_plots] added Boundary Bay figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,809 INFO [make_plots] added Campbell River figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,811 INFO [make_plots] added Cherry Point figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,812 INFO [make_plots] added Friday Harbor figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,813 INFO [make_plots] added Halfmoon Bay figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,814 INFO [make_plots] added Nanaimo figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,815 INFO [make_plots] added Neah Bay figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,816 INFO [make_plots] added New Westminster figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,818 INFO [make_plots] added Patricia Bay figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,819 INFO [make_plots] added Point Atkinson figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,820 INFO [make_plots] added Port Renfrew figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,821 INFO [make_plots] added Sand Heads figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,823 INFO [make_plots] added Sandy Cove figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,824 INFO [make_plots] added Squamish figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,825 INFO [make_plots] added Victoria figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,828 INFO [make_plots] added Woodwards Landing figure to 2025-07-12 NEMO forecast publish render list 2025-07-12 10:10:08,829 INFO [make_plots] starting to render nemo forecast publish 12jul25 figures 2025-07-12 10:12:14,397 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18567519 = 2025-07-12 01:45:20 UTC, 7.3% complete 2025-07-12 10:14:41,718 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 99, 22.9% complete 2025-07-12 10:17:14,393 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18567697 = 2025-07-12 03:44:00 UTC, 15.6% complete 2025-07-12 10:17:41,409 WARNING [make_plots] invalid station number for Boundary Bay station: None 2025-07-12 10:17:41,411 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2025-07-12 10:18:05,884 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:19:14,512 WARNING [make_plots] invalid station number for Halfmoon Bay station: None 2025-07-12 10:19:14,513 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay 2025-07-12 10:19:36,482 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:19:41,719 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 207, 47.9% complete 2025-07-12 10:20:18,105 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:20:38,317 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:20:56,695 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:20:58,480 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12jul25 2025-07-12 10:21:14,254 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:21:30,035 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:21:47,234 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:22:04,499 WARNING [make_plots] invalid station number for Squamish station: None 2025-07-12 10:22:04,500 WARNING [make_plots] can't resolve a valid CHS station code for Squamish 2025-07-12 10:22:14,394 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18567874 = 2025-07-12 05:42:00 UTC, 23.8% complete 2025-07-12 10:22:21,060 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:22:36,731 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-07-07 00:00:00Z to 2025-07-14 11:55:00Z 2025-07-12 10:22:53,679 INFO [make_plots] finished rendering nemo forecast publish 12jul25 figures 2025-07-12 10:22:53,682 INFO [make_plots] nemo publish plots for 2025-07-12 forecast completed 2025-07-12 10:22:53,691 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-07-12 10:22:53,861 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:22:57,256 INFO [make_feeds] running in process 1519370 2025-07-12 10:22:57,257 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:22:57,259 INFO [make_feeds] publishing log messages to tcp://*:5564 2025-07-12 10:22:57,260 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:23:11,834 INFO [make_feeds] ATOM feeds for 2025-07-12 forecast run completed 2025-07-12 10:23:11,837 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2025-07-12 10:24:41,719 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 309, 71.5% complete 2025-07-12 10:25:35,611 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/12jul25 2025-07-12 10:25:35,636 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-07-12 forecast into /results/nowcast-sys/figures/surface_currents/forecast/12jul25 2025-07-12 10:25:35,637 INFO [make_surface_current_tiles] surface current tile figures for 2025-07-12 forecast completed 2025-07-12 10:25:35,941 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2025-07-12 10:27:14,394 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18568052 = 2025-07-12 07:40:40 UTC, 32.0% complete 2025-07-12 10:29:41,719 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 417, 96.5% complete 2025-07-12 10:32:14,395 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18568230 = 2025-07-12 09:39:20 UTC, 40.3% complete 2025-07-12 10:34:41,720 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2025-07-12 10:34:41,723 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2025-07-12 10:34:42,191 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 10:34:42,198 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-12'], host='arbutus.cloud-nowcast') 2025-07-12 10:34:43,471 INFO [download_wwatch3_results] running in process 1519500 2025-07-12 10:34:43,472 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:34:43,473 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562 2025-07-12 10:34:43,475 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:34:44,154 INFO [run_ww3] running in process 18300 2025-07-12 10:34:44,156 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:34:44,157 INFO [run_ww3] publishing log messages to tcp://*:5572 2025-07-12 10:34:44,158 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:34:44,161 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-07-12T103444.153047-0700 2025-07-12 10:34:44,163 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-07-12T103444.153047-0700/SoGWW3.sh on arbutus.cloud-nowcast 2025-07-12 10:34:44,173 INFO [run_ww3] forecast WaveWatch3 run for 2025-07-12 on arbutus.cloud-nowcast started 2025-07-12 10:34:44,178 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2025-07-12 10:34:44,506 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2025-07-12 10:34:46,507 INFO [watch_ww3] running in process 18455 2025-07-12 10:34:46,509 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 10:34:46,510 INFO [watch_ww3] publishing log messages to tcp://*:5573 2025-07-12 10:34:46,511 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2025-07-12 10:34:46,543 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2025-07-12 10:35:13,800 INFO [download_wwatch3_results] nowcast 2025-07-12 results files from arbutus.cloud-nowcast downloaded 2025-07-12 10:35:13,803 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2025-07-12 10:37:14,396 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18568409 = 2025-07-12 11:38:40 UTC, 48.5% complete 2025-07-12 10:39:46,543 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete 2025-07-12 10:42:14,396 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18568586 = 2025-07-12 13:36:40 UTC, 56.7% complete 2025-07-12 10:44:46,543 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 207, 31.9% complete 2025-07-12 10:47:14,397 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18568765 = 2025-07-12 15:36:00 UTC, 65.0% complete 2025-07-12 10:49:46,544 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete 2025-07-12 10:52:14,399 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18568942 = 2025-07-12 17:34:00 UTC, 73.2% complete 2025-07-12 10:54:46,546 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 417, 64.4% complete 2025-07-12 10:57:14,400 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18569120 = 2025-07-12 19:32:40 UTC, 81.5% complete 2025-07-12 10:59:46,547 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 525, 81.0% complete 2025-07-12 11:02:14,401 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18569298 = 2025-07-12 21:31:20 UTC, 89.7% complete 2025-07-12 11:04:46,548 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 630, 97.2% complete 2025-07-12 11:07:14,403 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18569476 = 2025-07-12 23:30:00 UTC, 98.0% complete 2025-07-12 11:09:46,550 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2025-07-12 11:09:46,553 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2025-07-12 11:09:47,007 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 11:09:48,275 INFO [download_wwatch3_results] running in process 1520754 2025-07-12 11:09:48,276 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:09:48,277 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562 2025-07-12 11:09:48,279 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:10:26,927 INFO [download_wwatch3_results] forecast 2025-07-12 results files from arbutus.cloud-nowcast downloaded 2025-07-12 11:10:26,931 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2025-07-12 11:10:27,254 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 11:10:28,526 INFO [update_forecast_datasets] running in process 1520774 2025-07-12 11:10:28,528 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:10:28,529 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2025-07-12 11:10:28,530 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:10:28,531 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/12jul25 run 2025-07-12 11:10:28,535 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/12jul25 run 2025-07-12 11:10:28,536 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/12jul25 run 2025-07-12 11:10:28,545 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/12jul25 run: /results/SalishSea/rolling-forecasts/wwatch3 2025-07-12 11:10:28,546 INFO [update_forecast_datasets] wwatch3 2025-07-12 forecast rolling forecast datasets updated 2025-07-12 11:10:28,548 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2025-07-12 11:10:28,872 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2025-07-12 11:10:30,079 INFO [ping_erddap] running in process 1520779 2025-07-12 11:10:30,081 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:10:30,082 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-07-12 11:10:30,083 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:10:30,085 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2025-07-12 11:10:30,087 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-12 11:10:30,423 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 11:10:34,030 INFO [make_plots] running in process 1520785 2025-07-12 11:10:34,031 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:10:34,032 INFO [make_plots] publishing log messages to tcp://*:5562 2025-07-12 11:10:34,033 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:10:34,034 INFO [make_plots] preparing render list for 2025-07-12 WaveWatch3 forecast publish figures 2025-07-12 11:10:34,035 INFO [make_plots] added Halibut Bank figure to 2025-07-12 WaveWatch3 forecast publish render list 2025-07-12 11:10:34,036 INFO [make_plots] added Sentry Shoal figure to 2025-07-12 WaveWatch3 forecast publish render list 2025-07-12 11:10:34,037 INFO [make_plots] starting to render wwatch3 forecast publish 12jul25 figures 2025-07-12 11:10:34,486 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2025-07-12 11:10:40,676 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2025-07-12 11:10:45,886 INFO [make_plots] finished rendering wwatch3 forecast publish 12jul25 figures 2025-07-12 11:10:45,887 INFO [make_plots] wwatch3 publish plots for 2025-07-12 forecast completed 2025-07-12 11:10:45,891 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-07-12 11:12:14,404 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18569520 = 2025-07-12 23:59:20 UTC, 100.0% complete 2025-07-12 11:17:14,412 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2025-07-12 11:17:14,415 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2025-07-12 11:17:14,870 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 11:17:16,261 INFO [download_results] running in process 1520895 2025-07-12 11:17:16,263 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:17:16,264 INFO [download_results] publishing log messages to tcp://*:5562 2025-07-12 11:17:16,265 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:17:16,267 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/12jul25 to /results2/SalishSea/nowcast-green.202111 2025-07-12 11:19:54,183 INFO [download_results] nowcast-green 2025-07-12 results files from arbutus.cloud-nowcast downloaded 2025-07-12 11:19:54,189 INFO [manager] checklist updated with [results files] items from download_results worker 2025-07-12 11:19:54,529 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 11:19:54,534 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2025-07-12 11:19:54,539 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 11:19:54,545 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 11:19:54,550 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 11:19:55,774 INFO [ping_erddap] running in process 1520929 2025-07-12 11:19:55,775 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:19:55,776 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-07-12 11:19:55,777 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:19:55,782 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2025-07-12 11:19:55,783 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-12 11:19:56,912 INFO [make_averaged_dataset] running in process 1520932 2025-07-12 11:19:56,913 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:19:56,914 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562 2025-07-12 11:19:56,915 INFO [make_averaged_dataset] running in process 1520930 2025-07-12 11:19:56,916 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:19:56,917 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564 2025-07-12 11:19:56,918 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:19:56,918 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jul-2025 chemistry 2025-07-12 11:19:56,919 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:19:56,920 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jul-2025 biology 2025-07-12 11:19:56,929 INFO [make_averaged_dataset] running in process 1520931 2025-07-12 11:19:56,930 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:19:56,930 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565 2025-07-12 11:19:56,931 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:19:56,932 INFO [make_averaged_dataset] creating day-averaged dataset for 12-Jul-2025 physics 2025-07-12 11:19:57,560 INFO [make_plots] running in process 1520927 2025-07-12 11:19:57,561 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 11:19:57,562 INFO [make_plots] publishing log messages to tcp://*:5566 2025-07-12 11:19:57,563 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-07-12 11:19:57,899 INFO [make_plots] preparing render list for 2025-07-12 NEMO nowcast-green research figures 2025-07-12 11:19:59,761 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:00,528 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:01,284 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:02,045 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:02,806 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:03,561 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:04,312 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:05,192 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:06,092 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:06,948 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:07,809 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:08,595 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:09,594 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:09,597 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:09,598 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:09,599 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-07-12 NEMO nowcast-green research render list 2025-07-12 11:20:09,600 INFO [make_plots] starting to render nemo nowcast-green research 12jul25 figures 2025-07-12 11:20:32,300 INFO [make_averaged_dataset] day-averaged dataset for 12-Jul-2025 physics created 2025-07-12 11:20:32,303 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2025-07-12 11:21:07,958 INFO [make_averaged_dataset] day-averaged dataset for 12-Jul-2025 chemistry created 2025-07-12 11:21:07,961 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2025-07-12 11:21:31,282 INFO [make_averaged_dataset] day-averaged dataset for 12-Jul-2025 biology created 2025-07-12 11:21:31,286 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2025-07-12 12:22:54,377 INFO [make_plots] finished rendering nemo nowcast-green research 12jul25 figures 2025-07-12 12:22:54,546 INFO [make_plots] nemo research plots for 2025-07-12 nowcast-green completed 2025-07-12 12:22:54,610 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-07-12 15:04:05,846 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250712/18/ 2025-07-12 15:04:05,848 INFO [collect_weather] 2025-07-12 2.5km weather forecast 18 collection complete 2025-07-12 15:04:05,851 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2025-07-12 15:04:06,206 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250712/18/ 2025-07-12 15:04:06,207 INFO [crop_gribs] 2025-07-12 18 GRIBs cropping complete 2025-07-12 15:04:06,349 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2025-07-12 15:04:06,357 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2025-07-12 15:04:06,362 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2025-07-12 15:04:06,366 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-07-13'], host='localhost') 2025-07-12 15:04:06,373 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2025-07-12 15:04:07,684 INFO [download_weather] running in process 1523558 2025-07-12 15:04:07,686 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 15:04:07,687 INFO [download_weather] publishing log messages to tcp://*:5562 2025-07-12 15:04:07,687 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2025-07-12 15:04:07,688 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250712 2025-07-12 15:04:07,689 INFO [download_weather] running in process 1523560 2025-07-12 15:04:07,690 INFO [collect_weather] running in process 1523561 2025-07-12 15:04:07,693 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 15:04:07,694 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 15:04:07,695 INFO [download_weather] publishing log messages to tcp://*:5563 2025-07-12 15:04:07,696 INFO [collect_weather] publishing log messages to tcp://*:5564 2025-07-12 15:04:07,697 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2025-07-12 15:04:07,698 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2025-07-12 15:04:07,699 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250712 2025-07-12 15:04:07,723 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2025-07-12 15:04:08,800 INFO [crop_gribs] running in process 1523562 2025-07-12 15:04:08,802 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 15:04:08,803 INFO [crop_gribs] publishing log messages to tcp://*:5565 2025-07-12 15:04:08,804 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2025-07-12 15:04:08,805 INFO [crop_gribs] cropping 2025-07-13 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2025-07-12 15:04:08,820 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250713/00/ 2025-07-12 15:07:50,588 INFO [download_weather] 2025-07-12 1km weather forecast 00 downloads complete 2025-07-12 15:07:50,591 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2025-07-12 15:07:51,457 INFO [download_weather] 2025-07-12 1km weather forecast 12 downloads complete 2025-07-12 15:07:51,460 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2025-07-12 15:57:58,086 INFO [upload_forcing] running in process 1525871 2025-07-12 15:57:58,087 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 15:57:58,089 INFO [upload_forcing] publishing log messages to tcp://*:5562 2025-07-12 15:57:58,090 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-12 15:57:58,936 INFO [upload_forcing] turbidity 2025-07-12 forcing files upload to orcinus-nowcast-agrif completed 2025-07-12 15:57:58,941 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-12 15:57:59,294 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 15:58:00,638 INFO [make_forcing_links] running in process 1525898 2025-07-12 15:58:00,639 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 15:58:00,641 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2025-07-12 15:58:00,642 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-07-12 15:58:01,609 INFO [make_forcing_links] nowcast-agrif 2025-07-12 forcing file links on orcinus-nowcast-agrif created 2025-07-12 15:58:01,613 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-07-12 15:58:01,970 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 15:58:03,337 INFO [run_NEMO_agrif] running in process 1525918 2025-07-12 15:58:03,340 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 15:58:03,342 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5562 2025-07-12 15:58:03,343 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2025-07-12 15:58:29,182 INFO [run_NEMO_agrif] job id for 12jul25nowcast-agrif: 11447732.orca2.ibb 2025-07-12 15:58:29,184 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2025-07-12 queued on orcinus-nowcast-agrif 2025-07-12 15:58:29,189 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker 2025-07-12 15:58:29,523 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11447732.orca2.ibb'], host='localhost') 2025-07-12 15:58:30,904 INFO [watch_NEMO_agrif] running in process 1525948 2025-07-12 15:58:30,906 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 15:58:30,907 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5562 2025-07-12 15:58:30,908 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2025-07-12 15:58:31,819 INFO [watch_NEMO_agrif] watching 12jul25nowcast-agrif job 11447732 on orcinus-nowcast-agrif 2025-07-12 15:58:32,051 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 15:59:32,255 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:00:32,461 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:01:32,665 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:02:32,871 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:03:33,075 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:04:33,278 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:05:33,482 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:06:33,681 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:07:33,876 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:08:34,074 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:09:34,273 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:10:34,477 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:11:34,680 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:12:34,883 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:13:35,085 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:14:35,288 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:15:35,492 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:16:35,688 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:17:35,891 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:18:36,097 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:19:36,298 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:20:36,501 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:21:36,707 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:22:36,912 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:23:37,151 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:24:37,355 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:25:37,594 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:26:37,800 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:27:38,010 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif job 11447732 is queued on orcinus-nowcast-agrif 2025-07-12 16:28:38,936 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2025-07-12 16:33:39,426 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2025-07-12 16:38:39,849 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2025-07-12 16:43:40,335 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2025-07-12 16:48:40,758 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2025-07-12 16:53:41,337 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8588392 = 2025-07-12 02:34:00 UTC, 10.7% complete 2025-07-12 16:58:41,783 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8588643 = 2025-07-12 05:21:20 UTC, 22.3% complete 2025-07-12 17:03:42,309 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8588897 = 2025-07-12 08:10:40 UTC, 34.1% complete 2025-07-12 17:08:42,747 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8589152 = 2025-07-12 11:00:40 UTC, 45.9% complete 2025-07-12 17:13:43,185 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8589408 = 2025-07-12 13:51:20 UTC, 57.8% complete 2025-07-12 17:18:43,653 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8589662 = 2025-07-12 16:40:40 UTC, 69.5% complete 2025-07-12 17:23:44,119 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8589916 = 2025-07-12 19:30:00 UTC, 81.3% complete 2025-07-12 17:28:44,561 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8590170 = 2025-07-12 22:19:20 UTC, 93.1% complete 2025-07-12 17:33:44,985 INFO [watch_NEMO_agrif] 12jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8590320 = 2025-07-12 23:59:20 UTC, 100.0% complete 2025-07-12 17:38:45,205 INFO [watch_NEMO_agrif] NEMO AGRIF run 11447732.orca2.ibb on orcinus-nowcast-agrif completed 2025-07-12 17:38:45,210 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker 2025-07-12 17:38:45,761 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 17:38:47,049 INFO [download_results] running in process 1527010 2025-07-12 17:38:47,050 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 17:38:47,052 INFO [download_results] publishing log messages to tcp://*:5562 2025-07-12 17:38:47,053 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-07-12 17:38:47,054 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/12jul25 to /results/SalishSea/nowcast-agrif.201702 2025-07-12 17:39:10,573 INFO [download_results] nowcast-agrif 2025-07-12 results files from orcinus-nowcast-agrif downloaded 2025-07-12 17:39:10,581 INFO [manager] checklist updated with [results files] items from download_results worker 2025-07-12 17:39:10,913 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2025-07-12'], host='localhost') 2025-07-12 17:39:15,045 INFO [make_plots] running in process 1527026 2025-07-12 17:39:15,046 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 17:39:15,047 INFO [make_plots] publishing log messages to tcp://*:5562 2025-07-12 17:39:15,048 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-07-12 17:39:15,397 INFO [make_plots] preparing render list for 2025-07-12 NEMO nowcast-agrif research figures 2025-07-12 17:39:15,398 INFO [make_plots] added baynes_sound_surface figure to 2025-07-12 NEMO nowcast-agrif research render list 2025-07-12 17:39:15,399 INFO [make_plots] starting to render nemo nowcast-agrif research 12jul25 figures 2025-07-12 17:39:26,810 INFO [make_plots] finished rendering nemo nowcast-agrif research 12jul25 figures 2025-07-12 17:39:26,813 INFO [make_plots] nemo research plots for 2025-07-12 nowcast-agrif completed 2025-07-12 17:39:26,816 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-07-12 21:09:53,486 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250713/00/ 2025-07-12 21:09:53,487 INFO [collect_weather] 2025-07-13 2.5km weather forecast 00 collection complete 2025-07-12 21:09:53,490 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2025-07-12 21:09:54,054 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2025-07-12 21:09:54,062 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-07-13'], host='localhost') 2025-07-12 21:09:54,746 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250713/00/ 2025-07-12 21:09:54,747 INFO [crop_gribs] 2025-07-13 00 GRIBs cropping complete 2025-07-12 21:09:54,749 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2025-07-12 21:09:55,301 INFO [collect_weather] running in process 1529311 2025-07-12 21:09:55,302 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 21:09:55,303 INFO [collect_weather] publishing log messages to tcp://*:5562 2025-07-12 21:09:55,304 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2025-07-12 21:09:55,333 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2025-07-12 21:09:56,111 INFO [crop_gribs] running in process 1529313 2025-07-12 21:09:56,113 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-12 21:09:56,114 INFO [crop_gribs] publishing log messages to tcp://*:5563 2025-07-12 21:09:56,115 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2025-07-12 21:09:56,116 INFO [crop_gribs] cropping 2025-07-13 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2025-07-12 21:09:56,129 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250713/06/ 2025-07-13 03:06:43,050 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250713/06/ 2025-07-13 03:06:43,051 INFO [collect_weather] 2025-07-13 2.5km weather forecast 06 collection complete 2025-07-13 03:06:43,054 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2025-07-13 03:06:43,645 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,657 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,662 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,668 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,673 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,678 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,681 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,684 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,689 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,693 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,700 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,723 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,733 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,743 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,752 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2025-07-13 03:06:43,770 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2025-07-13 03:06:43,778 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2025-07-13 03:06:43,801 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-07-12'], host='localhost') 2025-07-13 03:06:43,819 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2025-07-13 03:06:43,830 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2025-07-13 03:06:43,844 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2025-07-13 03:06:43,876 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250713/06/ 2025-07-13 03:06:43,876 INFO [crop_gribs] 2025-07-13 06 GRIBs cropping complete 2025-07-13 03:06:43,881 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2025-07-13 03:06:45,355 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2025-07-13 03:06:46,828 INFO [collect_NeahBay_ssh] running in process 1534183 2025-07-13 03:06:46,829 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:46,830 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562 2025-07-13 03:06:46,831 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:46,832 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-07-13 2025-07-13 03:06:46,833 INFO [collect_weather] running in process 1534184 2025-07-13 03:06:46,834 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:46,835 INFO [collect_weather] publishing log messages to tcp://*:5563 2025-07-13 03:06:46,838 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:46,859 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2025-07-13 03:06:47,674 INFO [collect_river_data] running in process 1534166 2025-07-13 03:06:47,675 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,676 INFO [collect_river_data] publishing log messages to tcp://*:5564 2025-07-13 03:06:47,677 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,678 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-07-12 2025-07-13 03:06:47,679 INFO [collect_river_data] running in process 1534173 2025-07-13 03:06:47,681 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,682 INFO [collect_river_data] publishing log messages to tcp://*:5565 2025-07-13 03:06:47,683 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,684 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-07-12 2025-07-13 03:06:47,724 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2025-07-13 03:06:47,724 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-07-12 completed 2025-07-13 03:06:47,726 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:47,729 INFO [collect_river_data] running in process 1534167 2025-07-13 03:06:47,730 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,731 INFO [collect_river_data] publishing log messages to tcp://*:5566 2025-07-13 03:06:47,732 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,732 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-07-12 2025-07-13 03:06:47,734 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2025-07-13 03:06:47,735 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-07-12 completed 2025-07-13 03:06:47,756 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-07-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2025-07-13 03:06:47,756 INFO [collect_river_data] ECCC Englishman river data collection for 2025-07-12 completed 2025-07-13 03:06:47,773 INFO [collect_river_data] running in process 1534172 2025-07-13 03:06:47,774 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,775 INFO [collect_river_data] publishing log messages to tcp://*:5567 2025-07-13 03:06:47,776 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,776 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-07-12 2025-07-13 03:06:47,777 INFO [collect_river_data] running in process 1534163 2025-07-13 03:06:47,780 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,780 INFO [collect_river_data] publishing log messages to tcp://*:5568 2025-07-13 03:06:47,781 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,782 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-07-12 2025-07-13 03:06:47,814 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2025-07-13 03:06:47,815 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-07-12 completed 2025-07-13 03:06:47,841 INFO [collect_river_data] running in process 1534174 2025-07-13 03:06:47,841 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,842 INFO [collect_river_data] publishing log messages to tcp://*:5569 2025-07-13 03:06:47,843 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,844 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-07-12 2025-07-13 03:06:47,853 INFO [collect_river_data] running in process 1534176 2025-07-13 03:06:47,854 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,855 INFO [collect_river_data] publishing log messages to tcp://*:5570 2025-07-13 03:06:47,855 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,856 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-07-12 2025-07-13 03:06:47,857 INFO [collect_river_data] running in process 1534165 2025-07-13 03:06:47,858 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,859 INFO [collect_river_data] publishing log messages to tcp://*:5571 2025-07-13 03:06:47,860 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,861 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-07-12 2025-07-13 03:06:47,872 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-07-12 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2025-07-13 03:06:47,872 INFO [collect_river_data] ECCC Capilano river data collection for 2025-07-12 completed 2025-07-13 03:06:47,882 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2025-07-13 03:06:47,882 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-07-12 completed 2025-07-13 03:06:47,883 INFO [collect_river_data] running in process 1534171 2025-07-13 03:06:47,884 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,885 INFO [collect_river_data] publishing log messages to tcp://*:5572 2025-07-13 03:06:47,885 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,886 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-07-12 2025-07-13 03:06:47,887 INFO [collect_river_data] running in process 1534168 2025-07-13 03:06:47,888 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,889 INFO [collect_river_data] publishing log messages to tcp://*:5573 2025-07-13 03:06:47,890 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,891 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-07-12 2025-07-13 03:06:47,892 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2025-07-13 03:06:47,893 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-07-12 completed 2025-07-13 03:06:47,895 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2025-07-13 03:06:47,895 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-07-12 completed 2025-07-13 03:06:47,905 INFO [collect_river_data] running in process 1534178 2025-07-13 03:06:47,906 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,907 INFO [collect_river_data] publishing log messages to tcp://*:5574 2025-07-13 03:06:47,909 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,910 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-07-12 2025-07-13 03:06:47,914 INFO [collect_river_data] running in process 1534169 2025-07-13 03:06:47,914 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,916 INFO [collect_river_data] publishing log messages to tcp://*:5575 2025-07-13 03:06:47,916 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,917 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-07-12 2025-07-13 03:06:47,919 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/RobertsCreek_flow 2025-07-13 03:06:47,919 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-07-12 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2025-07-13 03:06:47,920 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-07-12 completed 2025-07-13 03:06:47,921 INFO [collect_river_data] ECCC Fraser river data collection for 2025-07-12 completed 2025-07-13 03:06:47,925 INFO [collect_river_data] running in process 1534175 2025-07-13 03:06:47,926 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,927 INFO [collect_river_data] publishing log messages to tcp://*:5576 2025-07-13 03:06:47,928 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,929 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-07-12 2025-07-13 03:06:47,938 INFO [collect_river_data] running in process 1534170 2025-07-13 03:06:47,939 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:47,941 INFO [collect_river_data] publishing log messages to tcp://*:5577 2025-07-13 03:06:47,942 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:47,943 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-07-12 2025-07-13 03:06:47,945 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2025-07-13 03:06:47,945 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-07-12 completed 2025-07-13 03:06:47,947 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2025-07-13 03:06:47,948 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-07-12 completed 2025-07-13 03:06:47,966 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2025-07-13 03:06:47,966 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-07-12 completed 2025-07-13 03:06:47,976 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-07-12 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2025-07-13 03:06:47,976 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-07-12 completed 2025-07-13 03:06:48,077 INFO [get_vfpa_hadcp] running in process 1534182 2025-07-13 03:06:48,078 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:48,079 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578 2025-07-13 03:06:48,079 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:48,080 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-07-12 2025-07-13 03:06:48,167 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:48,326 INFO [crop_gribs] running in process 1534185 2025-07-13 03:06:48,327 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:48,328 INFO [crop_gribs] publishing log messages to tcp://*:5579 2025-07-13 03:06:48,329 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:48,330 INFO [crop_gribs] cropping 2025-07-13 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2025-07-13 03:06:48,341 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250713/12/ 2025-07-13 03:06:48,461 INFO [get_onc_ctd] running in process 1534179 2025-07-13 03:06:48,462 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:48,463 INFO [get_onc_ctd] publishing log messages to tcp://*:5565 2025-07-13 03:06:48,464 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:48,464 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-07-12 2025-07-13 03:06:48,497 INFO [get_onc_ferry] running in process 1534181 2025-07-13 03:06:48,498 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:48,499 INFO [get_onc_ferry] publishing log messages to tcp://*:5590 2025-07-13 03:06:48,500 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:48,501 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-07-12 2025-07-13 03:06:48,525 INFO [get_onc_ctd] running in process 1534180 2025-07-13 03:06:48,525 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:48,526 INFO [get_onc_ctd] publishing log messages to tcp://*:5591 2025-07-13 03:06:48,527 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:48,528 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-07-12 2025-07-13 03:06:48,544 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:48,627 INFO [grib_to_netcdf] running in process 1534410 2025-07-13 03:06:48,628 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:48,628 INFO [grib_to_netcdf] publishing log messages to tcp://*:5592 2025-07-13 03:06:48,629 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:48,630 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-07-13 forecast2 run 2025-07-13 03:06:48,890 INFO [collect_NeahBay_ssh] 2025-07-13T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2025-07-13 03:06:48,903 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:49,223 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:49,553 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:49,882 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:50,216 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:50,238 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-07-12 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202507.nc 2025-07-13 03:06:50,239 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-07 netcdf file 2025-07-13 03:06:50,528 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:50,840 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:51,168 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:51,496 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:51,862 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:52,254 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2025-07-13 03:06:52,632 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2025-07-13 03:06:52,971 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-07-13'], host='localhost') 2025-07-13 03:06:52,977 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2025-07-13 03:06:53,522 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2025-07-13 03:06:54,651 INFO [ping_erddap] running in process 1534530 2025-07-13 03:06:54,652 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:54,653 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-07-13 03:06:54,654 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:54,655 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2025-07-13 03:06:54,656 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-13 03:06:56,297 INFO [make_ssh_files] running in process 1534517 2025-07-13 03:06:56,298 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:06:56,299 INFO [make_ssh_files] publishing log messages to tcp://*:5562 2025-07-13 03:06:56,300 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:06:56,301 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-07-13 NOAA Neah Bay 00Z observation and forecast values 2025-07-13 03:06:58,127 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d14.nc 2025-07-13 03:06:58,680 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d08.nc 2025-07-13 03:06:58,940 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d09.nc 2025-07-13 03:06:59,192 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d10.nc 2025-07-13 03:06:59,458 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d11.nc 2025-07-13 03:06:59,662 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d12.nc 2025-07-13 03:06:59,881 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d13.nc 2025-07-13 03:07:00,081 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d14.nc 2025-07-13 03:07:00,259 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d15.nc 2025-07-13 03:07:00,463 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d16.nc 2025-07-13 03:07:00,591 INFO [make_ssh_files] sea surface height boundary file for 2025-07-13 forecast2 run created 2025-07-13 03:07:00,595 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2025-07-13 03:07:00,958 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost') 2025-07-13 03:07:00,965 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost') 2025-07-13 03:07:01,255 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d15.nc 2025-07-13 03:07:01,264 INFO [grib_to_netcdf] 2025-07-13 NEMO-atmos forcing file for forecast2 created 2025-07-13 03:07:01,266 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2025-07-13 03:07:02,375 INFO [make_201702_runoff_file] running in process 1534565 2025-07-13 03:07:02,377 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:02,378 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5562 2025-07-13 03:07:02,379 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:02,407 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry 2025-07-13 03:07:02,490 INFO [make_201702_runoff_file] 2025-07-13 runoff file creation from Fraser at Hope and climatology elsewhere complete 2025-07-13 03:07:02,493 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker 2025-07-13 03:07:02,949 INFO [make_runoff_file] running in process 1534563 2025-07-13 03:07:02,950 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:02,951 INFO [make_runoff_file] publishing log messages to tcp://*:5564 2025-07-13 03:07:02,952 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:02,955 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-07-12 2025-07-13 03:07:03,988 ERROR [make_runoff_file] no 2025-07-12 discharge obs for SkagitMountVernon: patching 2025-07-13 03:07:04,624 ERROR [make_runoff_file] no 2025-07-12 discharge obs for SnohomishMonroe: patching 2025-07-13 03:07:04,874 ERROR [make_runoff_file] no 2025-07-12 discharge obs for NisquallyMcKenna: patching 2025-07-13 03:07:04,984 ERROR [make_runoff_file] no 2025-07-12 discharge obs for GreenwaterGreenwater: patching 2025-07-13 03:07:06,358 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-07-12: /results/forcing/rivers/R202108Dailies_y2025m07d12.nc 2025-07-13 03:07:06,359 INFO [make_runoff_file] 2025-07-12 runoff file creation completed 2025-07-13 03:07:06,360 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2025-07-13 03:07:06,734 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2025-07-13 03:07:06,740 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2025-07-13 03:07:06,745 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2025-07-13 03:07:06,750 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2025-07-13 03:07:07,087 INFO [get_onc_ctd] 2025-07-12 ONC SEVIP CTD T&S file created 2025-07-13 03:07:07,089 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2025-07-13 03:07:07,501 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2025-07-13 03:07:07,777 INFO [get_onc_ctd] 2025-07-12 ONC SCVIP CTD T&S file created 2025-07-13 03:07:07,780 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2025-07-13 03:07:08,005 INFO [upload_forcing] running in process 1534600 2025-07-13 03:07:08,006 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:08,007 INFO [upload_forcing] publishing log messages to tcp://*:5562 2025-07-13 03:07:08,008 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:08,032 INFO [upload_forcing] running in process 1534598 2025-07-13 03:07:08,033 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:08,035 INFO [upload_forcing] publishing log messages to tcp://*:5564 2025-07-13 03:07:08,036 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:08,060 INFO [upload_forcing] running in process 1534599 2025-07-13 03:07:08,061 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:08,062 INFO [upload_forcing] publishing log messages to tcp://*:5566 2025-07-13 03:07:08,063 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:08,102 INFO [upload_forcing] running in process 1534601 2025-07-13 03:07:08,103 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:08,104 INFO [upload_forcing] publishing log messages to tcp://*:5567 2025-07-13 03:07:08,105 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:08,136 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost') 2025-07-13 03:07:08,652 INFO [ping_erddap] running in process 1534603 2025-07-13 03:07:08,653 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:08,654 INFO [ping_erddap] publishing log messages to tcp://*:5565 2025-07-13 03:07:08,655 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:08,657 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2025-07-13 03:07:08,659 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-13 03:07:09,265 INFO [ping_erddap] running in process 1534619 2025-07-13 03:07:09,266 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:09,268 INFO [ping_erddap] publishing log messages to tcp://*:5565 2025-07-13 03:07:09,269 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:09,272 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created 2025-07-13 03:07:09,274 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-13 03:07:09,335 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2025-07-12 2025-07-13 03:07:09,684 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d12.nc 2025-07-13 03:07:09,713 INFO [upload_forcing] forecast2 2025-07-13 forcing files upload to optimum-hindcast completed 2025-07-13 03:07:09,716 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-13 03:07:09,722 INFO [upload_forcing] forecast2 2025-07-13 forcing files upload to orcinus-nowcast-agrif completed 2025-07-13 03:07:10,037 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-13 03:07:11,882 INFO [upload_forcing] forecast2 2025-07-13 forcing files upload to arbutus.cloud-nowcast completed 2025-07-13 03:07:11,887 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-13 03:07:12,260 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-13'], host='localhost') 2025-07-13 03:07:13,739 INFO [make_forcing_links] running in process 1534641 2025-07-13 03:07:13,741 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:13,741 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2025-07-13 03:07:13,742 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:14,913 INFO [make_forcing_links] forecast2 2025-07-13 forcing file links on arbutus.cloud-nowcast created 2025-07-13 03:07:14,917 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2025-07-13 03:07:15,303 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-13'], host='arbutus.cloud-nowcast') 2025-07-13 03:07:16,798 INFO [upload_forcing] forecast2 2025-07-13 forcing files upload to robot.graham completed 2025-07-13 03:07:16,803 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2025-07-13 03:07:17,342 INFO [run_NEMO] running in process 27007 2025-07-13 03:07:17,344 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:17,345 INFO [run_NEMO] publishing log messages to tcp://*:5556 2025-07-13 03:07:19,849 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:20,609 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets 2025-07-13 03:07:20,917 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/12jul25forecast2_2025-07-13T030719.913089-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2025-07-13 03:07:20,929 INFO [run_NEMO] forecast2 NEMO run for 2025-07-13 on arbutus.cloud-nowcast started 2025-07-13 03:07:20,933 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2025-07-13 03:07:21,297 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2025-07-13 03:07:23,262 INFO [watch_NEMO] running in process 27280 2025-07-13 03:07:23,264 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:07:23,265 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2025-07-13 03:07:23,273 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:07:23,334 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2025-07-13 03:07:31,708 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2025-07-12 2025-07-13 03:07:53,735 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2025-07-12 2025-07-13 03:08:19,542 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2025-07-12 2025-07-13 03:08:19,581 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,586 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,588 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,589 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2025-07-12 2025-07-13 03:08:19,633 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,638 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,640 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,641 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2025-07-12 2025-07-13 03:08:19,685 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,688 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,689 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2025-07-12 2025-07-13 03:08:19,736 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,738 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,740 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2025-07-12 2025-07-13 03:08:19,783 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,786 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-12; substituting empty dataset 2025-07-13 03:08:19,979 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2025-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2025-07-13 03:08:19,995 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2025-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2025-07-13 03:08:20,011 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2025-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2025-07-13 03:08:20,027 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2025-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2025-07-13 03:08:20,043 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2025-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2025-07-13 03:08:20,060 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2025-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2025-07-13 03:08:20,075 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-12 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2025-07-13 03:09:34,096 INFO [get_onc_ferry] 2025-07-12 ONC TWDP ferry data file created 2025-07-13 03:09:34,099 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker 2025-07-13 03:09:34,389 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2025-07-13 03:09:35,557 INFO [ping_erddap] running in process 1534696 2025-07-13 03:09:35,559 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:09:35,560 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-07-13 03:09:35,561 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:09:35,563 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2025-07-13 03:09:35,566 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-13 03:12:23,331 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18572145 = 2025-07-14 05:09:20 UTC, 17.2% complete 2025-07-13 03:17:23,332 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18572682 = 2025-07-14 11:07:20 UTC, 37.1% complete 2025-07-13 03:22:23,333 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18573184 = 2025-07-14 16:42:00 UTC, 55.7% complete 2025-07-13 03:27:23,342 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18573691 = 2025-07-14 22:20:00 UTC, 74.5% complete 2025-07-13 03:32:23,335 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18574214 = 2025-07-15 04:08:40 UTC, 93.8% complete 2025-07-13 03:37:23,339 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2025-07-13 03:37:23,343 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2025-07-13 03:37:23,933 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-13'], host='arbutus.cloud-nowcast') 2025-07-13 03:37:23,965 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-13'], host='arbutus.cloud-nowcast') 2025-07-13 03:37:23,970 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-12'], host='localhost') 2025-07-13 03:37:25,291 INFO [download_results] running in process 1535041 2025-07-13 03:37:25,293 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:37:25,294 INFO [download_results] publishing log messages to tcp://*:5562 2025-07-13 03:37:25,296 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:37:25,297 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/12jul25 to /results/SalishSea/forecast2.202111 2025-07-13 03:37:26,679 INFO [make_ww3_current_file] running in process 27817 2025-07-13 03:37:26,681 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:37:26,682 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2025-07-13 03:37:26,683 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:37:26,684 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-07-13 forecast2 run 2025-07-13 03:37:26,787 INFO [make_ww3_wind_file] running in process 27819 2025-07-13 03:37:26,789 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:37:26,790 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2025-07-13 03:37:26,791 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:37:26,793 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-07-13 forecast2 run 2025-07-13 03:37:36,437 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-07-13 forecast2 run 2025-07-13 03:37:36,441 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2025-07-13 03:38:05,013 INFO [download_results] forecast2 2025-07-12 results files from arbutus.cloud-nowcast downloaded 2025-07-13 03:38:05,017 INFO [manager] checklist updated with [results files] items from download_results worker 2025-07-13 03:38:05,391 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-07-12'], host='localhost') 2025-07-13 03:38:07,602 INFO [make_CHS_currents_file] running in process 1535058 2025-07-13 03:38:07,604 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:38:07,605 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2025-07-13 03:38:07,606 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:38:47,050 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-12 for forecast2 2025-07-13 03:38:47,054 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2025-07-13 03:38:47,418 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-07-12'], host='localhost') 2025-07-13 03:38:48,626 INFO [update_forecast_datasets] running in process 1535079 2025-07-13 03:38:48,628 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:38:48,629 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2025-07-13 03:38:48,630 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:38:48,631 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/12jul25 run 2025-07-13 03:41:01,242 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-07-13 forecast2 run 2025-07-13 03:41:01,247 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2025-07-13 03:41:01,619 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-13'], host='arbutus.cloud-nowcast') 2025-07-13 03:41:03,618 INFO [run_ww3] running in process 28375 2025-07-13 03:41:03,619 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:41:03,620 INFO [run_ww3] publishing log messages to tcp://*:5572 2025-07-13 03:41:03,621 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:41:03,625 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-07-13T034103.608249-0700 2025-07-13 03:41:03,632 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2025-07-13 03:41:03,639 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-07-13T034103.608249-0700/SoGWW3.sh on arbutus.cloud-nowcast 2025-07-13 03:41:03,641 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-07-13 on arbutus.cloud-nowcast started 2025-07-13 03:41:03,965 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2025-07-13 03:41:05,973 INFO [watch_ww3] running in process 28462 2025-07-13 03:41:05,984 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:41:05,985 INFO [watch_ww3] publishing log messages to tcp://*:5573 2025-07-13 03:41:05,986 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:41:06,011 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2025-07-13 03:43:04,385 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/12jul25 run: /results/SalishSea/rolling-forecasts/nemo 2025-07-13 03:43:04,386 INFO [update_forecast_datasets] nemo 2025-07-12 forecast2 rolling forecast datasets updated 2025-07-13 03:43:04,389 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2025-07-13 03:43:04,763 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2025-07-13 03:43:04,768 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-07-12'], host='localhost') 2025-07-13 03:43:04,774 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-07-12'], host='localhost') 2025-07-13 03:43:06,143 INFO [ping_erddap] running in process 1535197 2025-07-13 03:43:06,145 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:43:06,146 INFO [ping_erddap] publishing log messages to tcp://*:5562 2025-07-13 03:43:06,148 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:43:06,160 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2025-07-13 03:43:06,162 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2025-07-13 03:43:07,435 INFO [make_surface_current_tiles] running in process 1535200 2025-07-13 03:43:07,436 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:43:07,437 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562 2025-07-13 03:43:07,437 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:43:07,440 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-07-12 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/12jul25 2025-07-13 03:43:07,734 INFO [make_plots] running in process 1535199 2025-07-13 03:43:07,735 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:43:07,736 INFO [make_plots] publishing log messages to tcp://*:5564 2025-07-13 03:43:07,737 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:43:08,108 INFO [make_plots] preparing render list for 2025-07-12 NEMO forecast2 publish figures 2025-07-13 03:43:08,205 INFO [make_plots] added Website_thumbnail figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,208 INFO [make_plots] added Threshold_website figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,209 INFO [make_plots] added PA_tidal_predictions figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,209 INFO [make_plots] added Boundary Bay figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,210 INFO [make_plots] added Campbell River figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,211 INFO [make_plots] added Cherry Point figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,214 INFO [make_plots] added Friday Harbor figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,215 INFO [make_plots] added Halfmoon Bay figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,216 INFO [make_plots] added Nanaimo figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,217 INFO [make_plots] added Neah Bay figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,218 INFO [make_plots] added New Westminster figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,219 INFO [make_plots] added Patricia Bay figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,220 INFO [make_plots] added Point Atkinson figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,220 INFO [make_plots] added Port Renfrew figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,221 INFO [make_plots] added Sand Heads figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,222 INFO [make_plots] added Sandy Cove figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,222 INFO [make_plots] added Squamish figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,223 INFO [make_plots] added Victoria figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,224 INFO [make_plots] added Woodwards Landing figure to 2025-07-12 NEMO forecast2 publish render list 2025-07-13 03:43:08,224 INFO [make_plots] starting to render nemo forecast2 publish 12jul25 figures 2025-07-13 03:46:06,010 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete 2025-07-13 03:50:46,483 WARNING [make_plots] invalid station number for Boundary Bay station: None 2025-07-13 03:50:46,484 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2025-07-13 03:51:06,018 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete 2025-07-13 03:51:10,538 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:52:22,709 WARNING [make_plots] invalid station number for Halfmoon Bay station: None 2025-07-13 03:52:22,710 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay 2025-07-13 03:52:46,127 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:53:27,397 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:53:51,592 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:54:15,792 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:54:31,741 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/12jul25 2025-07-13 03:54:33,263 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:54:48,743 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:55:05,937 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:55:22,846 WARNING [make_plots] invalid station number for Squamish station: None 2025-07-13 03:55:22,846 WARNING [make_plots] can't resolve a valid CHS station code for Squamish 2025-07-13 03:55:38,924 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:55:54,568 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-07-08 00:00:00Z to 2025-07-15 05:55:00Z 2025-07-13 03:56:06,013 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 306, 56.7% complete 2025-07-13 03:56:11,391 INFO [make_plots] finished rendering nemo forecast2 publish 12jul25 figures 2025-07-13 03:56:11,394 INFO [make_plots] nemo publish plots for 2025-07-12 forecast2 completed 2025-07-13 03:56:11,401 INFO [manager] checklist updated with [plots] items from make_plots worker 2025-07-13 03:56:11,709 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-07-12'], host='localhost') 2025-07-13 03:56:14,847 INFO [make_feeds] running in process 1537334 2025-07-13 03:56:14,848 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:56:14,849 INFO [make_feeds] publishing log messages to tcp://*:5564 2025-07-13 03:56:14,850 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:56:29,409 INFO [make_feeds] ATOM feeds for 2025-07-12 forecast2 run completed 2025-07-13 03:56:29,412 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2025-07-13 03:56:29,779 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2025-07-13 03:56:30,974 INFO [clear_checklist] running in process 1537357 2025-07-13 03:56:30,975 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:56:30,976 INFO [clear_checklist] publishing log messages to tcp://*:5564 2025-07-13 03:56:30,977 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:56:30,978 INFO [clear_checklist] requesting that manager clear system state checklist 2025-07-13 03:56:30,980 INFO [manager] writing checklist to log file 2025-07-13 03:56:31,027 INFO [manager] checklist cleared 2025-07-13 03:56:31,029 INFO [clear_checklist] nowcast system checklist cleared 2025-07-13 03:56:31,031 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2025-07-13 03:56:32,233 INFO [rotate_logs] running in process 1537363 2025-07-13 03:56:32,234 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2025-07-13 03:56:32,236 INFO [rotate_logs] publishing log messages to tcp://*:5564 2025-07-13 03:56:32,237 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2025-07-13 03:56:32,237 INFO [rotate_logs] rotating log files