2024-10-13 04:21:21,097 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2024-10-13 04:21:21,103 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2024-10-13 04:21:21,109 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2024-10-13 04:21:21,110 INFO [rotate_logs] log files rotated 2024-10-13 04:21:21,115 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2024-10-13 08:53:49,527 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20241013/12/ 2024-10-13 08:53:49,529 INFO [collect_weather] 2024-10-13 2.5km weather forecast 12 collection complete 2024-10-13 08:53:49,532 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-10-13 08:53:49,687 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-10-12'], host='localhost') 2024-10-13 08:53:49,694 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-10-12'], host='localhost') 2024-10-13 08:53:49,699 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-10-12'], host='localhost') 2024-10-13 08:53:49,703 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-10-12'], host='localhost') 2024-10-13 08:53:49,707 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2024-10-13 08:53:49,711 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2024-10-13 08:53:49,716 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2024-10-13 08:53:49,724 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2024-10-13 08:53:49,728 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2024-10-13 08:53:51,335 INFO [collect_weather] running in process 75496 2024-10-13 08:53:51,336 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:51,337 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-10-13 08:53:51,338 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:51,340 INFO [collect_NeahBay_ssh] running in process 75493 2024-10-13 08:53:51,341 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:51,342 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563 2024-10-13 08:53:51,343 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:51,343 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-10-13 2024-10-13 08:53:51,352 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2024-10-13 08:53:51,509 INFO [download_live_ocean] running in process 75494 2024-10-13 08:53:51,510 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:51,511 INFO [download_live_ocean] publishing log messages to tcp://*:5564 2024-10-13 08:53:51,512 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:51,512 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-10-13 2024-10-13 08:53:51,817 INFO [collect_river_data] running in process 75490 2024-10-13 08:53:51,818 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:51,819 INFO [collect_river_data] publishing log messages to tcp://*:5565 2024-10-13 08:53:51,820 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:51,821 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-10-12 2024-10-13 08:53:51,835 INFO [collect_river_data] running in process 75491 2024-10-13 08:53:51,836 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:51,837 INFO [collect_river_data] publishing log messages to tcp://*:5566 2024-10-13 08:53:51,838 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:51,839 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-10-12 2024-10-13 08:53:51,844 INFO [collect_river_data] running in process 75487 2024-10-13 08:53:51,845 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:51,846 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-10-13 08:53:51,847 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:51,847 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-10-12 2024-10-13 08:53:51,922 INFO [collect_river_data] running in process 75489 2024-10-13 08:53:51,923 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:51,925 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-10-13 08:53:51,926 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:51,927 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-10-12 2024-10-13 08:53:51,929 INFO [make_turbidity_file] running in process 75492 2024-10-13 08:53:51,931 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:51,932 INFO [make_turbidity_file] publishing log messages to tcp://*:5569 2024-10-13 08:53:51,933 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:51,934 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-10-13 2024-10-13 08:53:52,018 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2024-10-12&endDT=2024-10-12 "HTTP/1.1 200 OK" 2024-10-13 08:53:52,125 INFO [download_live_ocean] downloaded 10185439 bytes from https://liveocean.apl.uw.edu/output/f2024.10.13/ubc.nc to /results/forcing/LiveOcean/downloaded/20241013/low_passed_UBC.nc 2024-10-13 08:53:52,127 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2024-10-13 08:53:52,160 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2024-10-12&endDT=2024-10-12 "HTTP/1.1 200 OK" 2024-10-13 08:53:52,185 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2024-10-12&endDT=2024-10-12 "HTTP/1.1 200 OK" 2024-10-13 08:53:52,248 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2024-10-12&endDT=2024-10-12 "HTTP/1.1 200 OK" 2024-10-13 08:53:52,259 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-10-12 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2024-10-13 08:53:52,260 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-10-12 completed 2024-10-13 08:53:52,263 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-10-12 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2024-10-13 08:53:52,263 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-10-12 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2024-10-13 08:53:52,264 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-10-12 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2024-10-13 08:53:52,265 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-13 08:53:52,266 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-10-12 completed 2024-10-13 08:53:52,267 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-10-12 completed 2024-10-13 08:53:52,267 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-10-12 completed 2024-10-13 08:53:52,270 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-13 08:53:52,275 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-13 08:53:52,280 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-13 08:53:52,300 INFO [crop_gribs] running in process 75497 2024-10-13 08:53:52,301 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:52,302 INFO [crop_gribs] publishing log messages to tcp://*:5570 2024-10-13 08:53:52,303 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:52,304 INFO [crop_gribs] cropping 2024-10-13 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2024-10-13 08:53:52,325 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241013/18/ 2024-10-13 08:53:53,161 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20241013/low_passed_UBC.nc 2024-10-13 08:53:53,162 INFO [download_live_ocean] 2024-10-13 Live Ocean file for Salish Sea western boundary downloaded 2024-10-13 08:53:53,164 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2024-10-13 08:53:53,171 INFO [collect_NeahBay_ssh] 2024-10-13T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2024-10-13 08:53:53,319 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-10-13'], host='localhost') 2024-10-13 08:53:53,326 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-10-13 08:53:53,331 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 08:53:54,969 INFO [make_turbidity_file] 2024-10-13 Fraser River turbidity file creation complete 2024-10-13 08:53:54,971 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2024-10-13 08:53:55,366 INFO [make_live_ocean_files] running in process 75646 2024-10-13 08:53:55,367 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:55,368 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563 2024-10-13 08:53:55,369 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:55,381 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-10-13 Live Ocean run 2024-10-13 08:53:55,743 INFO [make_ssh_files] running in process 75648 2024-10-13 08:53:55,744 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:53:55,745 INFO [make_ssh_files] publishing log messages to tcp://*:5564 2024-10-13 08:53:55,746 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:53:55,747 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-10-13 NOAA Neah Bay 06Z observation and forecast values 2024-10-13 08:53:59,773 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d09.nc 2024-10-13 08:53:59,937 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d10.nc 2024-10-13 08:54:00,104 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d11.nc 2024-10-13 08:54:00,270 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d12.nc 2024-10-13 08:54:00,444 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d13.nc 2024-10-13 08:54:00,611 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d14.nc 2024-10-13 08:54:00,777 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d15.nc 2024-10-13 08:54:00,945 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d16.nc 2024-10-13 08:54:01,039 INFO [make_ssh_files] sea surface height boundary file for 2024-10-13 nowcast run created 2024-10-13 08:54:01,041 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-10-13 08:54:01,044 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-10-13 08:54:01,049 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-10-13 08:54:02,376 INFO [make_runoff_file] running in process 75705 2024-10-13 08:54:02,377 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:54:02,378 INFO [make_runoff_file] publishing log messages to tcp://*:5564 2024-10-13 08:54:02,379 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:54:02,402 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-10-13 08:54:02,479 INFO [make_runoff_file] 2024-10-13 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-10-13 08:54:02,481 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-10-13 08:54:02,803 INFO [make_v202111_runoff_file] running in process 75703 2024-10-13 08:54:02,804 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 08:54:02,805 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564 2024-10-13 08:54:02,806 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-10-13 08:54:02,807 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-10-12 2024-10-13 08:54:05,124 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-10-12: /results/forcing/rivers/R202108Dailies_y2024m10d12.nc 2024-10-13 08:54:05,126 INFO [make_v202111_runoff_file] 2024-10-12 runoff file creation completed 2024-10-13 08:54:05,129 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-10-13 08:54:34,717 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m10d13.nc 2024-10-13 08:54:34,719 INFO [make_live_ocean_files] 2024-10-13 Live Ocean western boundary conditions files created 2024-10-13 08:54:34,721 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2024-10-13 10:53:46,350 INFO [crop_gribs] crop_gribs 12 has watched for 8h; retrying remaining 1 file(s) 2024-10-13 10:53:48,519 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241013/12/ 2024-10-13 10:53:48,520 INFO [crop_gribs] 2024-10-13 12 GRIBs cropping complete 2024-10-13 10:53:48,522 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-10-13 10:53:48,671 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2024-10-13 10:53:50,513 INFO [grib_to_netcdf] running in process 76060 2024-10-13 10:53:50,513 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:53:50,515 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563 2024-10-13 10:53:50,516 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:53:50,517 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-10-13 nowcast and forecast runs 2024-10-13 10:53:58,074 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m10d13.nc 2024-10-13 10:54:04,722 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m10d14.nc 2024-10-13 10:54:08,447 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m10d15.nc 2024-10-13 10:54:08,461 INFO [grib_to_netcdf] 2024-10-13 NEMO-atmos forcing file for nowcast+ created 2024-10-13 10:54:08,464 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-10-13 10:54:08,472 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2024-10-13 10:54:08,476 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2024-10-13 10:54:08,479 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2024-10-13 10:54:08,485 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2024-10-13 10:54:08,489 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2024-10-13 10:54:09,617 INFO [ping_erddap] running in process 76102 2024-10-13 10:54:09,619 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:54:09,620 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-13 10:54:09,620 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:54:09,622 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2024-10-13 10:54:09,624 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-13 10:54:09,711 INFO [upload_forcing] running in process 76101 2024-10-13 10:54:09,712 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:54:09,713 INFO [upload_forcing] publishing log messages to tcp://*:5564 2024-10-13 10:54:09,714 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:54:09,736 INFO [upload_forcing] running in process 76097 2024-10-13 10:54:09,738 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:54:09,740 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-10-13 10:54:09,741 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:54:09,746 INFO [upload_forcing] running in process 76099 2024-10-13 10:54:09,747 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:54:09,748 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-10-13 10:54:09,749 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:54:09,755 INFO [upload_forcing] running in process 76100 2024-10-13 10:54:09,756 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:54:09,757 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-10-13 10:54:09,758 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:54:12,125 INFO [upload_forcing] nowcast+ 2024-10-13 forcing files upload to optimum-hindcast completed 2024-10-13 10:54:12,130 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-13 10:54:12,192 INFO [upload_forcing] nowcast+ 2024-10-13 forcing files upload to orcinus-nowcast-agrif completed 2024-10-13 10:54:12,196 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-13 10:54:19,463 INFO [upload_forcing] nowcast+ 2024-10-13 forcing files upload to robot.graham completed 2024-10-13 10:54:19,468 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-13 10:54:22,370 INFO [upload_forcing] nowcast+ 2024-10-13 forcing files upload to arbutus.cloud-nowcast completed 2024-10-13 10:54:22,375 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-13 10:54:22,383 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 10:54:23,496 INFO [make_forcing_links] running in process 76137 2024-10-13 10:54:23,498 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:54:23,499 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-10-13 10:54:23,500 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:54:24,642 INFO [make_forcing_links] nowcast+ 2024-10-13 forcing file links on arbutus.cloud-nowcast created 2024-10-13 10:54:24,647 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-13 10:54:24,655 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-10-13'], host='arbutus.cloud-nowcast') 2024-10-13 10:54:26,794 INFO [run_NEMO] running in process 5006 2024-10-13 10:54:26,795 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:54:26,797 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-10-13 10:54:29,618 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:54:29,649 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-10-13 10:54:29,651 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-10-13 10:54:29,658 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13oct24nowcast_2024-10-13T105426.845002-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-10-13 10:54:29,660 INFO [run_NEMO] nowcast NEMO run for 2024-10-13 on arbutus.cloud-nowcast started 2024-10-13 10:54:31,812 INFO [watch_NEMO] running in process 5261 2024-10-13 10:54:31,813 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 10:54:31,814 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-10-13 10:54:31,815 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-10-13 10:54:31,896 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-10-13 10:59:31,896 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17980267 = 2024-10-13 04:44:00 UTC, 19.7% complete 2024-10-13 11:04:31,897 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17980798 = 2024-10-13 10:38:00 UTC, 44.3% complete 2024-10-13 11:09:31,898 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17981293 = 2024-10-13 16:08:00 UTC, 67.3% complete 2024-10-13 11:14:31,899 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17981750 = 2024-10-13 21:12:40 UTC, 88.4% complete 2024-10-13 11:19:31,904 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2024-10-13 11:19:31,908 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-10-13 11:19:32,063 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:19:32,070 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:19:33,356 INFO [make_forcing_links] running in process 76196 2024-10-13 11:19:33,357 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:19:33,358 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-10-13 11:19:33,360 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:19:33,361 INFO [download_results] running in process 76197 2024-10-13 11:19:33,361 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:19:33,362 INFO [download_results] publishing log messages to tcp://*:5564 2024-10-13 11:19:33,363 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:19:33,364 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/13oct24 to /results/SalishSea/nowcast-blue.202111 2024-10-13 11:19:34,314 INFO [make_forcing_links] ssh 2024-10-13 forcing file links on arbutus.cloud-nowcast created 2024-10-13 11:19:34,318 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-13 11:19:34,329 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-13'], host='arbutus.cloud-nowcast') 2024-10-13 11:19:41,512 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/13oct24forecast_2024-10-13T111939.982124-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-10-13 11:19:41,514 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-10-13 11:19:41,526 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-10-13 11:19:41,897 INFO [run_NEMO] forecast NEMO run for 2024-10-13 on arbutus.cloud-nowcast started 2024-10-13 11:19:53,593 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-10-13 11:24:53,653 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17982454 = 2024-10-14 05:02:00 UTC, 14.0% complete 2024-10-13 11:29:53,638 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17982914 = 2024-10-14 10:08:40 UTC, 28.2% complete 2024-10-13 11:34:53,578 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17983433 = 2024-10-14 15:54:40 UTC, 44.2% complete 2024-10-13 11:39:39,526 INFO [download_results] nowcast 2024-10-13 results files from arbutus.cloud-nowcast downloaded 2024-10-13 11:39:39,530 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-13 11:39:39,540 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:39:39,546 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-10-12'], host='localhost') 2024-10-13 11:39:39,552 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:39:41,517 INFO [make_CHS_currents_file] running in process 76349 2024-10-13 11:39:41,518 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:39:41,519 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-10-13 11:39:41,519 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:39:42,931 INFO [make_plots] running in process 76348 2024-10-13 11:39:42,933 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:39:42,934 INFO [make_plots] publishing log messages to tcp://*:5564 2024-10-13 11:39:42,935 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:39:42,947 INFO [make_plots] running in process 76346 2024-10-13 11:39:42,976 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:39:42,983 INFO [make_plots] publishing log messages to tcp://*:5565 2024-10-13 11:39:42,984 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:39:43,615 INFO [make_plots] preparing render list for 2024-10-12 NEMO nowcast-blue comparison figures 2024-10-13 11:39:43,616 INFO [make_plots] preparing render list for 2024-10-13 NEMO nowcast-blue research figures 2024-10-13 11:39:44,924 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-blue research render list 2024-10-13 11:39:45,622 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-blue research render list 2024-10-13 11:39:45,623 INFO [make_plots] added Currents_sections_and_surface figure to 2024-10-13 NEMO nowcast-blue research render list 2024-10-13 11:39:45,624 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-10-13 NEMO nowcast-blue research render list 2024-10-13 11:39:45,625 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-10-13 NEMO nowcast-blue research render list 2024-10-13 11:39:45,626 INFO [make_plots] starting to render nemo nowcast research 13oct24 figures 2024-10-13 11:39:52,961 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17983931 = 2024-10-14 21:26:40 UTC, 59.6% complete 2024-10-13 11:40:03,584 INFO [make_plots] added SH_wind figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,585 INFO [make_plots] added Compare_VENUS_East figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,586 INFO [make_plots] added Compare_VENUS_Central figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,586 INFO [make_plots] added Central_ADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,587 INFO [make_plots] added Central_depavADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,588 INFO [make_plots] added Central_timeavADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,589 INFO [make_plots] added East_ADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,590 INFO [make_plots] added East_depavADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,590 INFO [make_plots] added East_timeavADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,591 INFO [make_plots] added ddl_ADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,592 INFO [make_plots] added ddl_depavADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,593 INFO [make_plots] added ddl_timeavADCP figure to 2024-10-12 NEMO nowcast-blue comparison render list 2024-10-13 11:40:03,594 INFO [make_plots] starting to render nemo nowcast comparison 12oct24 figures 2024-10-13 11:40:12,564 INFO [make_CHS_currents_file] Made CHS currents file for 2024-10-13 for nowcast 2024-10-13 11:40:12,566 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-10-13 11:40:55,715 INFO [make_plots] finished rendering nemo nowcast research 13oct24 figures 2024-10-13 11:40:55,718 INFO [make_plots] nemo research plots for 2024-10-13 nowcast completed 2024-10-13 11:40:55,739 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-13 11:41:29,169 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available 2024-10-13 11:41:29,687 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-10-13 11:41:30,180 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-10-13 11:41:30,669 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-10-13 11:41:31,161 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-10-13 11:41:31,650 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-10-13 11:41:32,138 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-10-13 11:41:32,630 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-10-13 11:41:33,120 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-10-13 11:41:33,608 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-10-13 11:41:33,609 INFO [make_plots] finished rendering nemo nowcast comparison 12oct24 figures 2024-10-13 11:41:33,688 INFO [make_plots] nemo comparison plots for 2024-10-12 nowcast completed 2024-10-13 11:41:33,692 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-13 11:44:52,964 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17984430 = 2024-10-15 02:59:20 UTC, 75.0% complete 2024-10-13 11:49:52,963 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17984931 = 2024-10-15 08:33:20 UTC, 90.5% complete 2024-10-13 11:54:52,967 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2024-10-13 11:54:52,972 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-10-13 11:54:53,154 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-13'], host='arbutus.cloud-nowcast') 2024-10-13 11:54:53,162 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-13'], host='arbutus.cloud-nowcast') 2024-10-13 11:54:53,166 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:54:53,172 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:54:53,176 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:54:53,181 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:54:53,185 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:54:54,485 INFO [upload_forcing] running in process 76481 2024-10-13 11:54:54,486 INFO [upload_forcing] running in process 76479 2024-10-13 11:54:54,487 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:54:54,488 INFO [upload_forcing] publishing log messages to tcp://*:5564 2024-10-13 11:54:54,489 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:54:54,490 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:54:54,492 INFO [upload_forcing] running in process 76478 2024-10-13 11:54:54,493 INFO [upload_forcing] running in process 76480 2024-10-13 11:54:54,493 INFO [upload_forcing] publishing log messages to tcp://*:5563 2024-10-13 11:54:54,495 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:54:54,495 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:54:54,496 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:54:54,497 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-10-13 11:54:54,498 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-10-13 11:54:54,499 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:54:54,500 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:54:54,502 INFO [download_results] running in process 76482 2024-10-13 11:54:54,503 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:54:54,505 INFO [download_results] publishing log messages to tcp://*:5567 2024-10-13 11:54:54,506 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:54:54,507 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/13oct24 to /results/SalishSea/forecast.202111 2024-10-13 11:54:55,110 INFO [upload_forcing] turbidity 2024-10-13 forcing files upload to optimum-hindcast completed 2024-10-13 11:54:55,114 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-13 11:54:55,742 INFO [upload_forcing] turbidity 2024-10-13 forcing files upload to arbutus.cloud-nowcast completed 2024-10-13 11:54:55,771 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-13 11:54:55,787 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 11:54:57,230 INFO [make_forcing_links] running in process 76510 2024-10-13 11:54:57,231 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 11:54:57,232 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-10-13 11:54:57,233 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-13 11:54:58,315 INFO [upload_forcing] turbidity 2024-10-13 forcing files upload to robot.graham completed 2024-10-13 11:54:58,319 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-13 11:55:11,371 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-10-13 forecast run 2024-10-13 11:55:11,374 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-10-13 11:55:41,618 INFO [make_forcing_links] nowcast-green 2024-10-13 forcing file links on arbutus.cloud-nowcast created 2024-10-13 11:55:41,622 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-13 11:55:41,640 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-10-13'], host='arbutus.cloud-nowcast') 2024-10-13 11:55:52,888 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-10-13 11:55:52,906 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2024-10-13 11:55:55,395 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-10-13 forecast run 2024-10-13 11:55:55,436 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-10-13 11:55:55,455 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-10-13'], host='arbutus.cloud-nowcast') 2024-10-13 11:56:03,503 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-10-13 11:56:05,300 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-10-13 11:56:05,317 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-10-13 11:56:13,806 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-13 11:57:04,582 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) TimeoutError: [Errno 110] Connection timed out 2024-10-13 12:01:03,584 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17980003 = 2024-10-13 01:48:00 UTC, 7.5% complete 2024-10-13 12:01:14,014 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete 2024-10-13 12:06:03,189 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17980180 = 2024-10-13 03:46:00 UTC, 15.7% complete 2024-10-13 12:06:14,011 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 183, 42.4% complete 2024-10-13 12:11:03,531 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17980356 = 2024-10-13 05:43:20 UTC, 23.9% complete 2024-10-13 12:11:14,033 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 282, 65.3% complete 2024-10-13 12:16:03,588 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17980533 = 2024-10-13 07:41:20 UTC, 32.1% complete 2024-10-13 12:16:14,014 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 378, 87.5% complete 2024-10-13 12:21:03,555 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17980710 = 2024-10-13 09:39:20 UTC, 40.3% complete 2024-10-13 12:21:14,015 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-10-13 12:21:14,018 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-10-13 12:21:14,193 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 12:21:14,200 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-13'], host='arbutus.cloud-nowcast') 2024-10-13 12:21:15,429 INFO [download_wwatch3_results] running in process 76657 2024-10-13 12:21:15,430 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:21:15,431 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563 2024-10-13 12:21:15,432 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:21:23,752 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-10-13 12:21:23,771 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-10-13 12:21:34,970 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-13 12:25:59,826 INFO [download_wwatch3_results] nowcast 2024-10-13 results files from arbutus.cloud-nowcast downloaded 2024-10-13 12:25:59,830 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-10-13 12:26:03,542 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17980886 = 2024-10-13 11:36:40 UTC, 48.4% complete 2024-10-13 12:26:30,911 INFO [download_results] forecast 2024-10-13 results files from arbutus.cloud-nowcast downloaded 2024-10-13 12:26:30,916 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-13 12:26:30,934 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 12:26:33,148 INFO [make_CHS_currents_file] running in process 76679 2024-10-13 12:26:33,150 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:26:33,151 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-10-13 12:26:33,152 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:26:34,338 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete 2024-10-13 12:27:13,976 INFO [make_CHS_currents_file] Made CHS currents file for 2024-10-13 for forecast 2024-10-13 12:27:13,979 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-10-13 12:27:13,998 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 12:27:15,060 INFO [update_forecast_datasets] running in process 76702 2024-10-13 12:27:15,062 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:27:15,063 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-10-13 12:27:15,064 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:27:15,065 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/13oct24 run 2024-10-13 12:27:15,314 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/13oct24 run: /results/SalishSea/rolling-forecasts/nemo 2024-10-13 12:27:15,315 INFO [update_forecast_datasets] nemo 2024-10-13 forecast rolling forecast datasets updated 2024-10-13 12:27:15,316 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-10-13 12:27:15,327 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-10-13 12:27:15,332 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 12:27:15,337 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 12:27:16,472 INFO [ping_erddap] running in process 76707 2024-10-13 12:27:16,474 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:27:16,475 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-13 12:27:16,475 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:27:16,483 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-10-13 12:27:16,484 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-13 12:27:17,564 INFO [make_surface_current_tiles] running in process 76710 2024-10-13 12:27:17,565 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:27:17,566 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2024-10-13 12:27:17,567 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:27:17,569 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-10-13 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/13oct24 2024-10-13 12:27:19,262 INFO [make_plots] running in process 76709 2024-10-13 12:27:19,263 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:27:19,264 INFO [make_plots] publishing log messages to tcp://*:5564 2024-10-13 12:27:19,265 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:27:19,881 INFO [make_plots] preparing render list for 2024-10-13 NEMO forecast publish figures 2024-10-13 12:27:20,002 INFO [make_plots] added Website_thumbnail figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,003 INFO [make_plots] added Threshold_website figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,004 INFO [make_plots] added PA_tidal_predictions figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,005 INFO [make_plots] added Boundary Bay figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,005 INFO [make_plots] added Campbell River figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,006 INFO [make_plots] added Cherry Point figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,007 INFO [make_plots] added Friday Harbor figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,008 INFO [make_plots] added Halfmoon Bay figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,009 INFO [make_plots] added Nanaimo figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,009 INFO [make_plots] added Neah Bay figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,010 INFO [make_plots] added New Westminster figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,011 INFO [make_plots] added Patricia Bay figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,011 INFO [make_plots] added Point Atkinson figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,012 INFO [make_plots] added Port Renfrew figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,012 INFO [make_plots] added Sand Heads figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,013 INFO [make_plots] added Sandy Cove figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,014 INFO [make_plots] added Squamish figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,014 INFO [make_plots] added Victoria figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,015 INFO [make_plots] added Woodwards Landing figure to 2024-10-13 NEMO forecast publish render list 2024-10-13 12:27:20,016 INFO [make_plots] starting to render nemo forecast publish 13oct24 figures 2024-10-13 12:31:02,901 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17981061 = 2024-10-13 13:33:20 UTC, 56.5% complete 2024-10-13 12:31:34,339 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 183, 28.2% complete 2024-10-13 12:36:02,903 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17981237 = 2024-10-13 15:30:40 UTC, 64.7% complete 2024-10-13 12:36:34,341 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 273, 42.1% complete 2024-10-13 12:37:49,314 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13oct24 2024-10-13 12:40:12,379 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-10-13 12:40:12,380 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-10-13 12:40:46,832 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:41:02,904 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17981413 = 2024-10-13 17:28:00 UTC, 72.8% complete 2024-10-13 12:41:34,342 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 372, 57.4% complete 2024-10-13 12:42:27,549 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/13oct24 2024-10-13 12:42:27,559 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-10-13 forecast into /results/nowcast-sys/figures/surface_currents/forecast/13oct24 2024-10-13 12:42:27,560 INFO [make_surface_current_tiles] surface current tile figures for 2024-10-13 forecast completed 2024-10-13 12:42:27,880 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-10-13 12:42:31,878 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:42:32,061 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-10-13 12:42:32,688 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:43:41,038 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:44:15,618 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:44:50,048 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:45:24,477 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:45:58,555 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:46:02,906 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17981589 = 2024-10-13 19:25:20 UTC, 81.0% complete 2024-10-13 12:46:32,968 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:46:34,343 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 462, 71.3% complete 2024-10-13 12:47:07,388 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:47:07,571 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-10-13 12:47:07,806 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:47:41,655 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-10-08 00:00:00Z to 2024-10-15 11:55:00Z 2024-10-13 12:48:15,733 INFO [make_plots] finished rendering nemo forecast publish 13oct24 figures 2024-10-13 12:48:15,734 INFO [make_plots] nemo publish plots for 2024-10-13 forecast completed 2024-10-13 12:48:15,742 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-13 12:48:15,906 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 12:48:18,930 INFO [make_feeds] running in process 78773 2024-10-13 12:48:18,932 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:48:18,933 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-10-13 12:48:18,935 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:48:47,471 INFO [make_feeds] ATOM feeds for 2024-10-13 forecast run completed 2024-10-13 12:48:47,473 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-10-13 12:51:02,907 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17981765 = 2024-10-13 21:22:40 UTC, 89.1% complete 2024-10-13 12:51:34,345 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 552, 85.2% complete 2024-10-13 12:53:48,872 INFO [upload_forcing] running in process 78801 2024-10-13 12:53:48,873 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:53:48,874 INFO [upload_forcing] publishing log messages to tcp://*:5563 2024-10-13 12:53:48,875 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:53:49,637 INFO [upload_forcing] turbidity 2024-10-13 forcing files upload to orcinus-nowcast-agrif completed 2024-10-13 12:53:49,642 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-13 12:53:49,779 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 12:53:50,926 INFO [make_forcing_links] running in process 78809 2024-10-13 12:53:50,927 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 12:53:50,928 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-10-13 12:53:50,930 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-13 12:56:01,482 CRITICAL [make_forcing_links] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_forcing_links.py", line 101, in make_forcing_links ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) TimeoutError: [Errno 110] Connection timed out 2024-10-13 12:56:02,908 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17981942 = 2024-10-13 23:20:40 UTC, 97.3% complete 2024-10-13 12:56:34,346 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-13 13:00:52,621 INFO [make_forcing_links] running in process 78836 2024-10-13 13:00:52,623 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:00:52,624 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-10-13 13:00:52,625 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:00:53,382 INFO [make_forcing_links] nowcast-agrif 2024-10-13 forcing file links on orcinus-nowcast-agrif created 2024-10-13 13:00:53,387 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-13 13:00:53,554 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:00:54,663 INFO [run_NEMO_agrif] running in process 78844 2024-10-13 13:00:54,664 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:00:54,666 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5563 2024-10-13 13:00:54,667 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:01:02,910 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17982000 = 2024-10-13 23:59:20 UTC, 100.0% complete 2024-10-13 13:01:06,758 INFO [run_NEMO_agrif] job id for 13oct24nowcast-agrif: 11410389.orca2.ibb 2024-10-13 13:01:06,759 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-10-13 queued on orcinus-nowcast-agrif 2024-10-13 13:01:06,764 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker 2024-10-13 13:01:06,926 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11410389.orca2.ibb'], host='localhost') 2024-10-13 13:01:08,063 INFO [watch_NEMO_agrif] running in process 78858 2024-10-13 13:01:08,065 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:01:08,066 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5563 2024-10-13 13:01:08,067 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:01:13,866 INFO [watch_NEMO_agrif] watching 13oct24nowcast-agrif job 11410389 on orcinus-nowcast-agrif 2024-10-13 13:01:14,099 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif job 11410389 is queued on orcinus-nowcast-agrif 2024-10-13 13:01:34,348 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-10-13 13:01:34,350 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-10-13 13:01:34,688 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:01:35,772 INFO [download_wwatch3_results] running in process 78868 2024-10-13 13:01:35,773 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:01:35,774 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564 2024-10-13 13:01:35,775 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:02:14,308 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif job 11410389 is queued on orcinus-nowcast-agrif 2024-10-13 13:03:15,221 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2024-10-13 13:04:10,439 INFO [download_wwatch3_results] forecast 2024-10-13 results files from arbutus.cloud-nowcast downloaded 2024-10-13 13:04:10,443 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-10-13 13:04:10,610 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:04:11,920 INFO [update_forecast_datasets] running in process 78887 2024-10-13 13:04:11,921 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:04:11,922 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2024-10-13 13:04:11,923 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:04:11,924 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/13oct24 run 2024-10-13 13:04:11,926 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/13oct24 run 2024-10-13 13:04:11,964 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/13oct24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-10-13 13:04:11,965 INFO [update_forecast_datasets] wwatch3 2024-10-13 forecast rolling forecast datasets updated 2024-10-13 13:04:11,967 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-10-13 13:04:12,130 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-10-13 13:04:13,096 INFO [ping_erddap] running in process 78892 2024-10-13 13:04:13,097 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:04:13,098 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-10-13 13:04:13,099 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:04:13,101 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-10-13 13:04:13,102 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-13 13:04:13,241 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:04:16,863 INFO [make_plots] running in process 78897 2024-10-13 13:04:16,864 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:04:16,865 INFO [make_plots] publishing log messages to tcp://*:5564 2024-10-13 13:04:16,866 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:04:16,867 INFO [make_plots] preparing render list for 2024-10-13 WaveWatch3 forecast publish figures 2024-10-13 13:04:16,868 INFO [make_plots] added Halibut Bank figure to 2024-10-13 WaveWatch3 forecast publish render list 2024-10-13 13:04:16,869 INFO [make_plots] added Sentry Shoal figure to 2024-10-13 WaveWatch3 forecast publish render list 2024-10-13 13:04:16,870 INFO [make_plots] starting to render wwatch3 forecast publish 13oct24 figures 2024-10-13 13:04:17,210 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-10-13 13:04:24,389 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-10-13 13:04:30,713 INFO [make_plots] finished rendering wwatch3 forecast publish 13oct24 figures 2024-10-13 13:04:30,714 INFO [make_plots] wwatch3 publish plots for 2024-10-13 forecast completed 2024-10-13 13:04:30,718 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-13 13:06:03,514 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2024-10-13 13:06:03,518 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-10-13 13:06:03,832 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:06:05,262 INFO [download_results] running in process 78978 2024-10-13 13:06:05,263 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:06:05,264 INFO [download_results] publishing log messages to tcp://*:5564 2024-10-13 13:06:05,266 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:06:05,267 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/13oct24 to /results2/SalishSea/nowcast-green.202111 2024-10-13 13:08:15,718 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8000863 = 2024-10-13 02:28:00 UTC, 10.3% complete 2024-10-13 13:13:16,167 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8001116 = 2024-10-13 05:16:40 UTC, 22.0% complete 2024-10-13 13:18:16,626 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8001371 = 2024-10-13 08:06:40 UTC, 33.8% complete 2024-10-13 13:23:17,114 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8001629 = 2024-10-13 10:58:40 UTC, 45.8% complete 2024-10-13 13:28:17,592 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8001886 = 2024-10-13 13:50:00 UTC, 57.7% complete 2024-10-13 13:33:18,080 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8002141 = 2024-10-13 16:40:00 UTC, 69.5% complete 2024-10-13 13:38:18,529 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8002397 = 2024-10-13 19:30:40 UTC, 81.3% complete 2024-10-13 13:39:22,469 INFO [download_results] nowcast-green 2024-10-13 results files from arbutus.cloud-nowcast downloaded 2024-10-13 13:39:22,476 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-13 13:39:22,647 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:39:22,653 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2024-10-13 13:39:22,658 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:39:22,663 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:39:22,667 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:39:23,990 INFO [ping_erddap] running in process 79106 2024-10-13 13:39:23,990 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:39:23,991 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-10-13 13:39:23,992 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:39:23,997 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2024-10-13 13:39:23,998 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-13 13:39:25,032 INFO [make_averaged_dataset] running in process 79107 2024-10-13 13:39:25,033 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:39:25,034 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564 2024-10-13 13:39:25,035 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:39:25,036 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Oct-2024 biology 2024-10-13 13:39:25,037 INFO [make_averaged_dataset] running in process 79108 2024-10-13 13:39:25,038 INFO [make_averaged_dataset] running in process 79109 2024-10-13 13:39:25,039 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:39:25,040 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:39:25,041 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565 2024-10-13 13:39:25,041 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566 2024-10-13 13:39:25,042 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:39:25,043 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:39:25,044 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Oct-2024 chemistry 2024-10-13 13:39:25,044 INFO [make_averaged_dataset] creating day-averaged dataset for 13-Oct-2024 physics 2024-10-13 13:39:26,173 INFO [make_plots] running in process 79104 2024-10-13 13:39:26,174 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:39:26,175 INFO [make_plots] publishing log messages to tcp://*:5567 2024-10-13 13:39:26,176 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:39:26,909 INFO [make_plots] preparing render list for 2024-10-13 NEMO nowcast-green research figures 2024-10-13 13:39:28,668 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:29,401 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:30,128 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:30,865 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:31,618 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:32,342 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:33,116 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:33,955 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:34,802 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:35,614 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:36,437 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:37,195 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:37,880 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:37,880 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:37,881 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:37,882 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-10-13 NEMO nowcast-green research render list 2024-10-13 13:39:37,882 INFO [make_plots] starting to render nemo nowcast-green research 13oct24 figures 2024-10-13 13:39:54,777 INFO [make_averaged_dataset] day-averaged dataset for 13-Oct-2024 physics created 2024-10-13 13:39:54,781 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-10-13 13:40:20,476 INFO [make_averaged_dataset] day-averaged dataset for 13-Oct-2024 chemistry created 2024-10-13 13:40:20,480 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-10-13 13:40:33,348 INFO [make_averaged_dataset] day-averaged dataset for 13-Oct-2024 biology created 2024-10-13 13:40:33,351 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-10-13 13:43:19,012 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8002651 = 2024-10-13 22:20:00 UTC, 93.1% complete 2024-10-13 13:48:19,440 INFO [watch_NEMO_agrif] 13oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8002800 = 2024-10-13 23:59:20 UTC, 100.0% complete 2024-10-13 13:53:19,663 INFO [watch_NEMO_agrif] NEMO AGRIF run 11410389.orca2.ibb on orcinus-nowcast-agrif completed 2024-10-13 13:53:19,667 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker 2024-10-13 13:53:20,014 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:53:21,787 INFO [download_results] running in process 79264 2024-10-13 13:53:21,791 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:53:21,792 INFO [download_results] publishing log messages to tcp://*:5563 2024-10-13 13:53:21,793 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:53:21,794 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/13oct24 to /results/SalishSea/nowcast-agrif.201702 2024-10-13 13:53:47,127 INFO [download_results] nowcast-agrif 2024-10-13 results files from orcinus-nowcast-agrif downloaded 2024-10-13 13:53:47,135 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-13 13:53:47,308 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-10-13'], host='localhost') 2024-10-13 13:53:52,162 INFO [make_plots] running in process 79271 2024-10-13 13:53:52,165 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 13:53:52,166 INFO [make_plots] publishing log messages to tcp://*:5563 2024-10-13 13:53:52,167 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-13 13:53:52,740 INFO [make_plots] preparing render list for 2024-10-13 NEMO nowcast-agrif research figures 2024-10-13 13:53:52,742 INFO [make_plots] added baynes_sound_surface figure to 2024-10-13 NEMO nowcast-agrif research render list 2024-10-13 13:53:52,742 INFO [make_plots] starting to render nemo nowcast-agrif research 13oct24 figures 2024-10-13 13:54:02,618 INFO [make_plots] finished rendering nemo nowcast-agrif research 13oct24 figures 2024-10-13 13:54:02,621 INFO [make_plots] nemo research plots for 2024-10-13 nowcast-agrif completed 2024-10-13 13:54:02,625 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-13 14:42:45,984 INFO [make_plots] finished rendering nemo nowcast-green research 13oct24 figures 2024-10-13 14:42:46,152 INFO [make_plots] nemo research plots for 2024-10-13 nowcast-green completed 2024-10-13 14:42:46,218 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-13 14:54:47,047 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20241013/18/ 2024-10-13 14:54:47,049 INFO [collect_weather] 2024-10-13 2.5km weather forecast 18 collection complete 2024-10-13 14:54:47,051 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-10-13 14:54:47,612 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2024-10-13 14:54:47,619 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2024-10-13 14:54:47,624 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2024-10-13 14:54:47,630 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-10-14'], host='localhost') 2024-10-13 14:54:48,299 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241013/18/ 2024-10-13 14:54:48,300 INFO [crop_gribs] 2024-10-13 18 GRIBs cropping complete 2024-10-13 14:54:48,303 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-10-13 14:54:48,902 INFO [download_weather] running in process 79495 2024-10-13 14:54:48,903 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 14:54:48,904 INFO [download_weather] publishing log messages to tcp://*:5562 2024-10-13 14:54:48,905 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-13 14:54:48,906 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20241013 2024-10-13 14:54:48,907 INFO [download_weather] running in process 79497 2024-10-13 14:54:48,908 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 14:54:48,908 INFO [download_weather] publishing log messages to tcp://*:5563 2024-10-13 14:54:48,909 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-13 14:54:48,910 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20241013 2024-10-13 14:54:48,914 INFO [collect_weather] running in process 79498 2024-10-13 14:54:48,916 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 14:54:48,916 INFO [collect_weather] publishing log messages to tcp://*:5564 2024-10-13 14:54:48,917 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-13 14:54:48,952 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2024-10-13 14:54:49,762 INFO [crop_gribs] running in process 79499 2024-10-13 14:54:49,763 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 14:54:49,765 INFO [crop_gribs] publishing log messages to tcp://*:5565 2024-10-13 14:54:49,766 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-10-13 14:54:49,767 INFO [crop_gribs] cropping 2024-10-14 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2024-10-13 14:54:49,780 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241014/00/ 2024-10-13 14:58:22,352 INFO [download_weather] 2024-10-13 1km weather forecast 00 downloads complete 2024-10-13 14:58:22,355 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-10-13 14:58:24,199 INFO [download_weather] 2024-10-13 1km weather forecast 12 downloads complete 2024-10-13 14:58:24,202 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-10-13 20:54:52,683 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20241014/00/ 2024-10-13 20:54:52,684 INFO [collect_weather] 2024-10-14 2.5km weather forecast 00 collection complete 2024-10-13 20:54:52,687 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-10-13 20:54:53,034 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2024-10-13 20:54:53,043 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-10-14'], host='localhost') 2024-10-13 20:54:54,336 INFO [collect_weather] running in process 81888 2024-10-13 20:54:54,337 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 20:54:54,338 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-10-13 20:54:54,339 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-13 20:54:54,354 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2024-10-13 20:54:54,595 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241014/00/ 2024-10-13 20:54:54,596 INFO [crop_gribs] 2024-10-14 00 GRIBs cropping complete 2024-10-13 20:54:54,599 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-10-13 20:54:55,097 INFO [crop_gribs] running in process 81890 2024-10-13 20:54:55,099 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-13 20:54:55,100 INFO [crop_gribs] publishing log messages to tcp://*:5563 2024-10-13 20:54:55,101 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-10-13 20:54:55,102 INFO [crop_gribs] cropping 2024-10-14 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2024-10-13 20:54:55,117 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241014/06/ 2024-10-14 02:54:11,071 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20241014/06/ 2024-10-14 02:54:11,072 INFO [collect_weather] 2024-10-14 2.5km weather forecast 06 collection complete 2024-10-14 02:54:11,075 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-10-14 02:54:11,430 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,440 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,445 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,449 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,454 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,460 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,463 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,466 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,470 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,474 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,479 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,483 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,492 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,499 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,511 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2024-10-14 02:54:11,539 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2024-10-14 02:54:11,550 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2024-10-14 02:54:11,582 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-10-13'], host='localhost') 2024-10-14 02:54:11,619 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2024-10-14 02:54:11,650 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2024-10-14 02:54:11,661 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2024-10-14 02:54:12,251 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241014/06/ 2024-10-14 02:54:12,252 INFO [crop_gribs] 2024-10-14 06 GRIBs cropping complete 2024-10-14 02:54:12,254 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-10-14 02:54:12,845 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2024-10-14 02:54:14,603 INFO [collect_weather] running in process 83324 2024-10-14 02:54:14,607 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:14,608 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-10-14 02:54:14,609 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:14,628 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2024-10-14 02:54:14,640 INFO [collect_NeahBay_ssh] running in process 83323 2024-10-14 02:54:14,641 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:14,642 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563 2024-10-14 02:54:14,643 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:14,644 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-10-14 2024-10-14 02:54:15,562 INFO [collect_river_data] running in process 83310 2024-10-14 02:54:15,563 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,564 INFO [collect_river_data] publishing log messages to tcp://*:5564 2024-10-14 02:54:15,565 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,565 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-10-13 2024-10-14 02:54:15,569 INFO [collect_river_data] running in process 83318 2024-10-14 02:54:15,570 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,571 INFO [collect_river_data] publishing log messages to tcp://*:5565 2024-10-14 02:54:15,571 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,572 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-10-13 2024-10-14 02:54:15,576 INFO [collect_river_data] running in process 83316 2024-10-14 02:54:15,577 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,579 INFO [collect_river_data] publishing log messages to tcp://*:5566 2024-10-14 02:54:15,580 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,580 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-10-13 2024-10-14 02:54:15,592 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2024-10-14 02:54:15,593 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-10-13 completed 2024-10-14 02:54:15,594 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:15,605 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2024-10-14 02:54:15,606 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-10-13 completed 2024-10-14 02:54:15,615 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2024-10-14 02:54:15,616 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-10-13 completed 2024-10-14 02:54:15,671 INFO [collect_river_data] running in process 83312 2024-10-14 02:54:15,672 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,673 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-10-14 02:54:15,674 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,675 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-10-13 2024-10-14 02:54:15,687 INFO [collect_river_data] running in process 83304 2024-10-14 02:54:15,688 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,689 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-10-14 02:54:15,689 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,690 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-10-13 2024-10-14 02:54:15,691 INFO [collect_river_data] running in process 83311 2024-10-14 02:54:15,692 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,694 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-10-14 02:54:15,694 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,695 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-10-13 2024-10-14 02:54:15,708 INFO [collect_river_data] running in process 83308 2024-10-14 02:54:15,710 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/RobertsCreek_flow 2024-10-14 02:54:15,710 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-10-13 completed 2024-10-14 02:54:15,711 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,712 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-10-14 02:54:15,713 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,713 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-10-13 2024-10-14 02:54:15,727 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2024-10-14 02:54:15,728 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-10-13 completed 2024-10-14 02:54:15,732 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-10-13 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2024-10-14 02:54:15,732 INFO [collect_river_data] ECCC Capilano river data collection for 2024-10-13 completed 2024-10-14 02:54:15,737 INFO [collect_river_data] running in process 83309 2024-10-14 02:54:15,738 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,739 INFO [collect_river_data] publishing log messages to tcp://*:5571 2024-10-14 02:54:15,740 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,741 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-10-13 2024-10-14 02:54:15,744 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-10-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2024-10-14 02:54:15,745 INFO [collect_river_data] ECCC Englishman river data collection for 2024-10-13 completed 2024-10-14 02:54:15,747 INFO [collect_river_data] running in process 83315 2024-10-14 02:54:15,747 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,748 INFO [collect_river_data] publishing log messages to tcp://*:5572 2024-10-14 02:54:15,749 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,750 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-10-13 2024-10-14 02:54:15,750 INFO [collect_river_data] running in process 83307 2024-10-14 02:54:15,751 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,753 INFO [collect_river_data] publishing log messages to tcp://*:5573 2024-10-14 02:54:15,753 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,754 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-10-13 2024-10-14 02:54:15,773 INFO [collect_river_data] running in process 83317 2024-10-14 02:54:15,775 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,776 INFO [collect_river_data] publishing log messages to tcp://*:5574 2024-10-14 02:54:15,777 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-10-13 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2024-10-14 02:54:15,782 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2024-10-14 02:54:15,782 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,783 INFO [collect_river_data] ECCC Fraser river data collection for 2024-10-13 completed 2024-10-14 02:54:15,784 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-10-13 completed 2024-10-14 02:54:15,785 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-10-13 2024-10-14 02:54:15,786 INFO [collect_river_data] running in process 83306 2024-10-14 02:54:15,787 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,788 INFO [collect_river_data] publishing log messages to tcp://*:5575 2024-10-14 02:54:15,789 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2024-10-14 02:54:15,790 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,790 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-10-13 completed 2024-10-14 02:54:15,791 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-10-13 2024-10-14 02:54:15,801 INFO [collect_river_data] running in process 83314 2024-10-14 02:54:15,801 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,803 INFO [collect_river_data] publishing log messages to tcp://*:5576 2024-10-14 02:54:15,804 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,805 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-10-13 2024-10-14 02:54:15,806 INFO [collect_river_data] running in process 83313 2024-10-14 02:54:15,811 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:15,812 INFO [collect_river_data] publishing log messages to tcp://*:5577 2024-10-14 02:54:15,813 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2024-10-14 02:54:15,813 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:15,814 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-10-13 completed 2024-10-14 02:54:15,815 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-10-13 2024-10-14 02:54:15,827 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2024-10-14 02:54:15,829 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-10-13 completed 2024-10-14 02:54:15,838 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2024-10-14 02:54:15,839 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-10-13 completed 2024-10-14 02:54:15,841 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-10-13 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2024-10-14 02:54:15,841 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-10-13 completed 2024-10-14 02:54:15,892 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:16,038 INFO [collect_NeahBay_ssh] 2024-10-14T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2024-10-14 02:54:16,044 INFO [get_vfpa_hadcp] running in process 83322 2024-10-14 02:54:16,045 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:16,045 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578 2024-10-14 02:54:16,046 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:16,046 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-10-13 2024-10-14 02:54:16,149 INFO [crop_gribs] running in process 83325 2024-10-14 02:54:16,152 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:16,153 INFO [crop_gribs] publishing log messages to tcp://*:5564 2024-10-14 02:54:16,154 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:16,155 INFO [crop_gribs] cropping 2024-10-14 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2024-10-14 02:54:16,157 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:16,172 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241014/12/ 2024-10-14 02:54:16,280 INFO [get_onc_ctd] running in process 83319 2024-10-14 02:54:16,282 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:16,283 INFO [get_onc_ctd] publishing log messages to tcp://*:5579 2024-10-14 02:54:16,283 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:16,284 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-10-13 2024-10-14 02:54:16,325 INFO [get_onc_ferry] running in process 83321 2024-10-14 02:54:16,326 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:16,327 INFO [get_onc_ferry] publishing log messages to tcp://*:5590 2024-10-14 02:54:16,328 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:16,329 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-10-13 2024-10-14 02:54:16,344 INFO [get_onc_ctd] running in process 83320 2024-10-14 02:54:16,345 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:16,346 INFO [get_onc_ctd] publishing log messages to tcp://*:5591 2024-10-14 02:54:16,347 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:16,347 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-10-13 2024-10-14 02:54:16,354 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:16,459 INFO [grib_to_netcdf] running in process 83518 2024-10-14 02:54:16,460 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:16,461 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565 2024-10-14 02:54:16,462 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:16,463 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-10-14 forecast2 run 2024-10-14 02:54:16,546 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:16,755 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:16,937 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:17,128 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:17,311 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:17,514 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:17,710 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:17,720 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-10-13 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202410.nc 2024-10-14 02:54:17,721 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-10 netcdf file 2024-10-14 02:54:17,888 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:18,060 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:18,267 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-14 02:54:18,438 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-10-14 02:54:18,653 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-10-14'], host='localhost') 2024-10-14 02:54:18,661 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2024-10-14 02:54:18,924 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2024-10-14 02:54:19,918 INFO [ping_erddap] running in process 83789 2024-10-14 02:54:19,919 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:19,920 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-14 02:54:19,921 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:19,923 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2024-10-14 02:54:19,925 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-14 02:54:21,111 INFO [make_ssh_files] running in process 83776 2024-10-14 02:54:21,112 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:21,113 INFO [make_ssh_files] publishing log messages to tcp://*:5563 2024-10-14 02:54:21,114 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:21,115 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-10-14 NOAA Neah Bay 00Z observation and forecast values 2024-10-14 02:54:21,344 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-10-13 2024-10-14 02:54:21,805 CRITICAL [get_onc_ctd] 2024-10-13 ONC SCVIP CTD T&S file creation failed 2024-10-14 02:54:24,804 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m10d15.nc 2024-10-14 02:54:25,707 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d09.nc 2024-10-14 02:54:25,872 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d10.nc 2024-10-14 02:54:26,040 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d11.nc 2024-10-14 02:54:26,209 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d12.nc 2024-10-14 02:54:26,375 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d13.nc 2024-10-14 02:54:26,574 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d14.nc 2024-10-14 02:54:26,749 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d15.nc 2024-10-14 02:54:26,932 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d16.nc 2024-10-14 02:54:27,060 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m10d16.nc 2024-10-14 02:54:27,066 INFO [grib_to_netcdf] 2024-10-14 NEMO-atmos forcing file for forecast2 created 2024-10-14 02:54:27,069 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-10-14 02:54:27,108 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d17.nc 2024-10-14 02:54:27,221 INFO [make_ssh_files] sea surface height boundary file for 2024-10-14 forecast2 run created 2024-10-14 02:54:27,274 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-10-14 02:54:27,471 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-10-14 02:54:27,478 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-10-14 02:54:28,727 INFO [make_runoff_file] running in process 83836 2024-10-14 02:54:28,728 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:28,729 INFO [make_runoff_file] publishing log messages to tcp://*:5563 2024-10-14 02:54:28,730 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:28,757 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-10-14 02:54:28,829 INFO [make_runoff_file] 2024-10-14 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-10-14 02:54:28,832 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-10-14 02:54:29,395 INFO [make_v202111_runoff_file] running in process 83834 2024-10-14 02:54:29,396 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:29,397 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5563 2024-10-14 02:54:29,398 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:29,399 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-10-13 2024-10-14 02:54:30,318 ERROR [make_v202111_runoff_file] no 2024-10-13 discharge obs for Skagit_MountVernon: patching 2024-10-14 02:54:30,927 ERROR [make_v202111_runoff_file] no 2024-10-13 discharge obs for Snohomish_Monroe: patching 2024-10-14 02:54:31,125 ERROR [make_v202111_runoff_file] no 2024-10-13 discharge obs for Nisqually_McKenna: patching 2024-10-14 02:54:31,208 ERROR [make_v202111_runoff_file] no 2024-10-13 discharge obs for Greenwater_Greenwater: patching 2024-10-14 02:54:32,451 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-10-13: /results/forcing/rivers/R202108Dailies_y2024m10d13.nc 2024-10-14 02:54:32,452 INFO [make_v202111_runoff_file] 2024-10-13 runoff file creation completed 2024-10-14 02:54:32,454 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-10-14 02:54:32,652 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2024-10-14 02:54:32,658 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2024-10-14 02:54:32,663 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2024-10-14 02:54:32,669 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2024-10-14 02:54:33,906 INFO [upload_forcing] running in process 83877 2024-10-14 02:54:33,908 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:33,908 INFO [upload_forcing] publishing log messages to tcp://*:5563 2024-10-14 02:54:33,909 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:33,935 INFO [upload_forcing] running in process 83875 2024-10-14 02:54:33,936 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:33,937 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-10-14 02:54:33,938 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:33,958 INFO [upload_forcing] running in process 83876 2024-10-14 02:54:33,960 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:33,961 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-10-14 02:54:33,961 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:33,977 INFO [upload_forcing] running in process 83874 2024-10-14 02:54:33,978 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:33,979 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-10-14 02:54:33,980 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:35,159 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m10d13.nc 2024-10-14 02:54:35,179 INFO [upload_forcing] forecast2 2024-10-14 forcing files upload to optimum-hindcast completed 2024-10-14 02:54:35,183 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-14 02:54:35,982 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-10-13 2024-10-14 02:54:36,788 INFO [upload_forcing] forecast2 2024-10-14 forcing files upload to arbutus.cloud-nowcast completed 2024-10-14 02:54:36,792 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-14 02:54:36,996 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-14'], host='localhost') 2024-10-14 02:54:38,250 INFO [make_forcing_links] running in process 83902 2024-10-14 02:54:38,251 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:38,253 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-10-14 02:54:38,254 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:38,652 WARNING [get_onc_ctd] no SEVIP salinity data 2024-10-14 02:54:38,882 INFO [get_onc_ctd] 2024-10-13 ONC SEVIP CTD T&S file created 2024-10-14 02:54:38,885 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2024-10-14 02:54:39,081 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2024-10-14 02:54:39,420 INFO [make_forcing_links] forecast2 2024-10-14 forcing file links on arbutus.cloud-nowcast created 2024-10-14 02:54:39,423 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-14 02:54:39,622 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-14'], host='arbutus.cloud-nowcast') 2024-10-14 02:54:40,205 INFO [ping_erddap] running in process 83909 2024-10-14 02:54:40,207 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:40,208 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-14 02:54:40,209 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:40,210 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2024-10-14 02:54:40,212 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-14 02:54:41,518 INFO [upload_forcing] forecast2 2024-10-14 forcing files upload to robot.graham completed 2024-10-14 02:54:41,523 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-14 02:54:42,662 INFO [run_NEMO] running in process 14882 2024-10-14 02:54:42,663 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:42,664 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-10-14 02:54:42,792 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:44,866 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/13oct24forecast2_2024-10-14T025442.857729-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-10-14 02:54:44,922 INFO [run_NEMO] forecast2 NEMO run for 2024-10-14 on arbutus.cloud-nowcast started 2024-10-14 02:54:44,927 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-10-14 02:54:45,128 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-10-14 02:54:47,105 INFO [watch_NEMO] running in process 15137 2024-10-14 02:54:47,106 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:54:47,107 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-10-14 02:54:47,108 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:54:47,180 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-10-14 02:54:50,377 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-10-14 02:54:50,722 ERROR [upload_forcing] Traceback (most recent call last): 2024-10-14 02:54:50,723 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2327, in _check_banner 2024-10-14 02:54:50,725 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-10-14 02:54:50,726 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14 02:54:50,727 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 381, in readline 2024-10-14 02:54:50,729 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-10-14 02:54:50,730 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-14 02:54:50,731 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 626, in _read_timeout 2024-10-14 02:54:50,733 ERROR [upload_forcing] raise socket.timeout() 2024-10-14 02:54:50,735 ERROR [upload_forcing] TimeoutError 2024-10-14 02:54:50,736 ERROR [upload_forcing] 2024-10-14 02:54:50,738 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-10-14 02:54:50,739 ERROR [upload_forcing] 2024-10-14 02:54:50,741 ERROR [upload_forcing] Traceback (most recent call last): 2024-10-14 02:54:50,742 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2143, in run 2024-10-14 02:54:50,744 ERROR [upload_forcing] self._check_banner() 2024-10-14 02:54:50,745 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2331, in _check_banner 2024-10-14 02:54:50,747 ERROR [upload_forcing] raise SSHException( 2024-10-14 02:54:50,749 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-10-14 02:54:50,751 ERROR [upload_forcing] 2024-10-14 02:54:50,757 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 730, in _auth key = self._key_from_filepath( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 638, in _key_from_filepath key = klass.from_private_key_file(key_path, password) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/pkey.py", line 421, in from_private_key_file key = cls(filename=filename, password=password) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/ed25519key.py", line 60, in __init__ pkformat, data = self._read_private_key("OPENSSH", f) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/pkey.py", line 529, in _read_private_key raise SSHException( paramiko.ssh_exception.SSHException: encountered RSA key, expected OPENSSH key During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 85, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 451, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 738, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2143, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2331, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-10-14 02:55:01,686 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-10-13 2024-10-14 02:55:25,528 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-10-13 2024-10-14 02:55:49,366 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-10-13 2024-10-14 02:56:06,465 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-10-13 2024-10-14 02:56:23,202 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-10-13 2024-10-14 02:56:32,017 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-10-13 2024-10-14 02:56:40,167 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-10-13; substituting empty dataset 2024-10-14 02:56:40,168 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-10-13 2024-10-14 02:56:48,496 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-10-13; substituting empty dataset 2024-10-14 02:56:48,867 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-10-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-10-14 02:56:48,884 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-10-13 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-10-14 02:58:19,720 INFO [get_onc_ferry] 2024-10-13 ONC TWDP ferry data file created 2024-10-14 02:58:19,723 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker 2024-10-14 02:58:19,918 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2024-10-14 02:58:21,158 INFO [ping_erddap] running in process 83929 2024-10-14 02:58:21,159 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 02:58:21,160 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-14 02:58:21,161 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-14 02:58:21,163 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2024-10-14 02:58:21,165 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-14 02:59:47,181 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17984570 = 2024-10-15 04:32:40 UTC, 15.2% complete 2024-10-14 03:04:47,182 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17985047 = 2024-10-15 09:50:40 UTC, 32.8% complete 2024-10-14 03:09:47,184 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17985527 = 2024-10-15 15:10:40 UTC, 50.6% complete 2024-10-14 03:14:47,185 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17986031 = 2024-10-15 20:46:40 UTC, 69.3% complete 2024-10-14 03:19:47,185 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17986513 = 2024-10-16 02:08:00 UTC, 87.1% complete 2024-10-14 03:24:47,186 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17986860 = 2024-10-16 05:59:20 UTC, 100.0% complete 2024-10-14 03:29:47,193 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2024-10-14 03:29:47,197 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-10-14 03:29:47,545 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-14'], host='arbutus.cloud-nowcast') 2024-10-14 03:29:47,553 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-14'], host='arbutus.cloud-nowcast') 2024-10-14 03:29:47,558 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-13'], host='localhost') 2024-10-14 03:29:49,006 INFO [download_results] running in process 83998 2024-10-14 03:29:49,008 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 03:29:49,009 INFO [download_results] publishing log messages to tcp://*:5563 2024-10-14 03:29:49,010 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-14 03:29:49,011 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/13oct24 to /results/SalishSea/forecast2.202111 2024-10-14 03:30:19,257 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-10-14 forecast2 run 2024-10-14 03:30:19,259 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-10-14 03:32:14,286 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-10-14 forecast2 run 2024-10-14 03:32:14,289 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-10-14 03:32:14,489 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-14'], host='arbutus.cloud-nowcast') 2024-10-14 03:32:24,571 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-10-14 03:32:24,779 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-10-14 03:32:33,443 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-14 03:37:33,666 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 81, 15.0% complete 2024-10-14 03:42:33,681 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 165, 30.6% complete 2024-10-14 03:47:33,703 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 261, 48.3% complete 2024-10-14 03:52:33,673 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 354, 65.6% complete 2024-10-14 03:54:40,020 INFO [download_results] forecast2 2024-10-13 results files from arbutus.cloud-nowcast downloaded 2024-10-14 03:54:40,026 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-14 03:54:40,225 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-10-13'], host='localhost') 2024-10-14 03:54:42,212 INFO [make_CHS_currents_file] running in process 84168 2024-10-14 03:54:42,214 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 03:54:42,215 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-10-14 03:54:42,216 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-10-14 03:55:16,475 INFO [make_CHS_currents_file] Made CHS currents file for 2024-10-13 for forecast2 2024-10-14 03:55:16,478 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-10-14 03:55:16,679 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-10-13'], host='localhost') 2024-10-14 03:55:17,737 INFO [update_forecast_datasets] running in process 84195 2024-10-14 03:55:17,738 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 03:55:17,740 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-10-14 03:55:17,741 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-10-14 03:55:17,742 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/13oct24 run 2024-10-14 03:57:33,003 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 444, 82.2% complete 2024-10-14 03:59:27,343 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/13oct24 run: /results/SalishSea/rolling-forecasts/nemo 2024-10-14 03:59:27,344 INFO [update_forecast_datasets] nemo 2024-10-13 forecast2 rolling forecast datasets updated 2024-10-14 03:59:27,346 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-10-14 03:59:27,546 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-10-14 03:59:27,552 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-10-13'], host='localhost') 2024-10-14 03:59:27,557 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-10-13'], host='localhost') 2024-10-14 03:59:28,895 INFO [ping_erddap] running in process 84273 2024-10-14 03:59:28,896 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 03:59:28,897 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-14 03:59:28,898 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-14 03:59:28,907 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-10-14 03:59:28,908 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-14 03:59:29,965 INFO [make_surface_current_tiles] running in process 84276 2024-10-14 03:59:29,966 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 03:59:29,967 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2024-10-14 03:59:29,968 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-10-14 03:59:29,970 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-10-13 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/13oct24 2024-10-14 03:59:31,559 INFO [make_plots] running in process 84275 2024-10-14 03:59:31,560 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 03:59:31,561 INFO [make_plots] publishing log messages to tcp://*:5565 2024-10-14 03:59:31,563 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-14 03:59:32,163 INFO [make_plots] preparing render list for 2024-10-13 NEMO forecast2 publish figures 2024-10-14 03:59:32,290 INFO [make_plots] added Website_thumbnail figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,292 INFO [make_plots] added Threshold_website figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,292 INFO [make_plots] added PA_tidal_predictions figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,293 INFO [make_plots] added Boundary Bay figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,294 INFO [make_plots] added Campbell River figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,295 INFO [make_plots] added Cherry Point figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,295 INFO [make_plots] added Friday Harbor figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,296 INFO [make_plots] added Halfmoon Bay figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,297 INFO [make_plots] added Nanaimo figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,297 INFO [make_plots] added Neah Bay figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,298 INFO [make_plots] added New Westminster figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,299 INFO [make_plots] added Patricia Bay figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,300 INFO [make_plots] added Point Atkinson figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,300 INFO [make_plots] added Port Renfrew figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,301 INFO [make_plots] added Sand Heads figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,302 INFO [make_plots] added Sandy Cove figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,303 INFO [make_plots] added Squamish figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,304 INFO [make_plots] added Victoria figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,305 INFO [make_plots] added Woodwards Landing figure to 2024-10-13 NEMO forecast2 publish render list 2024-10-14 03:59:32,306 INFO [make_plots] starting to render nemo forecast2 publish 13oct24 figures 2024-10-14 04:02:33,573 CRITICAL [watch_ww3] unhandled exception: Traceback (most recent call last): File "/nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/nowcast/workers/watch_ww3.py", line 94, in watch_ww3 time_step = int(line.split()[0]) ~~~~~~~~~~~~^^^ IndexError: list index out of range 2024-10-14 04:10:42,709 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13oct24 2024-10-14 04:13:03,023 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-10-14 04:13:03,024 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-10-14 04:13:37,691 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:15:01,169 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/13oct24 2024-10-14 04:15:01,180 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-10-13 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/13oct24 2024-10-14 04:15:01,181 INFO [make_surface_current_tiles] surface current tile figures for 2024-10-13 forecast2 completed 2024-10-14 04:15:01,456 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-10-14 04:15:22,542 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:15:22,723 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-10-14 04:15:23,366 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:16:31,574 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:17:06,195 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:17:49,233 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:18:23,960 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:18:58,589 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:19:33,122 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:20:07,921 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:20:08,098 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-10-14 04:20:08,363 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:20:42,132 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-10-09 00:00:00Z to 2024-10-16 05:55:00Z 2024-10-14 04:21:16,654 INFO [make_plots] finished rendering nemo forecast2 publish 13oct24 figures 2024-10-14 04:21:16,655 INFO [make_plots] nemo publish plots for 2024-10-13 forecast2 completed 2024-10-14 04:21:16,663 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-14 04:21:16,995 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-10-13'], host='localhost') 2024-10-14 04:21:20,198 INFO [make_feeds] running in process 86274 2024-10-14 04:21:20,200 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 04:21:20,201 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-10-14 04:21:20,202 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-10-14 04:21:47,963 INFO [make_feeds] ATOM feeds for 2024-10-13 forecast2 run completed 2024-10-14 04:21:47,966 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-10-14 04:21:48,271 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2024-10-14 04:21:49,348 INFO [clear_checklist] running in process 86297 2024-10-14 04:21:49,350 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 04:21:49,351 INFO [clear_checklist] publishing log messages to tcp://*:5563 2024-10-14 04:21:49,352 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2024-10-14 04:21:49,353 INFO [clear_checklist] requesting that manager clear system state checklist 2024-10-14 04:21:49,355 INFO [manager] writing checklist to log file 2024-10-14 04:21:49,394 INFO [manager] checklist cleared 2024-10-14 04:21:49,396 INFO [clear_checklist] nowcast system checklist cleared 2024-10-14 04:21:49,398 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2024-10-14 04:21:50,439 INFO [rotate_logs] running in process 86302 2024-10-14 04:21:50,440 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-14 04:21:50,441 INFO [rotate_logs] publishing log messages to tcp://*:5563 2024-10-14 04:21:50,442 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2024-10-14 04:21:50,443 INFO [rotate_logs] rotating log files