2024-10-11 04:16:15,508 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2024-10-11 04:16:15,515 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2024-10-11 04:16:15,520 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2024-10-11 04:16:15,521 INFO [rotate_logs] log files rotated 2024-10-11 04:16:15,527 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2024-10-11 08:54:47,848 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20241011/12/ 2024-10-11 08:54:47,849 INFO [collect_weather] 2024-10-11 2.5km weather forecast 12 collection complete 2024-10-11 08:54:47,852 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-10-11 08:54:48,001 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-10-10'], host='localhost') 2024-10-11 08:54:48,007 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-10-10'], host='localhost') 2024-10-11 08:54:48,012 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-10-10'], host='localhost') 2024-10-11 08:54:48,017 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-10-10'], host='localhost') 2024-10-11 08:54:48,021 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2024-10-11 08:54:48,026 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2024-10-11 08:54:48,030 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2024-10-11 08:54:48,035 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2024-10-11 08:54:48,038 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2024-10-11 08:54:48,349 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241011/12/ 2024-10-11 08:54:48,350 INFO [crop_gribs] 2024-10-11 12 GRIBs cropping complete 2024-10-11 08:54:48,352 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-10-11 08:54:48,501 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2024-10-11 08:54:49,653 INFO [collect_NeahBay_ssh] running in process 50507 2024-10-11 08:54:49,655 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:49,655 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562 2024-10-11 08:54:49,656 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:49,658 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-10-11 2024-10-11 08:54:49,659 INFO [collect_weather] running in process 50509 2024-10-11 08:54:49,660 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:49,661 INFO [collect_weather] publishing log messages to tcp://*:5563 2024-10-11 08:54:49,662 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:49,682 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2024-10-11 08:54:49,869 INFO [download_live_ocean] running in process 50508 2024-10-11 08:54:49,870 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:49,871 INFO [download_live_ocean] publishing log messages to tcp://*:5564 2024-10-11 08:54:49,872 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:49,873 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-10-11 2024-10-11 08:54:50,242 INFO [collect_river_data] running in process 50501 2024-10-11 08:54:50,243 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:50,244 INFO [collect_river_data] publishing log messages to tcp://*:5565 2024-10-11 08:54:50,244 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:50,245 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-10-10 2024-10-11 08:54:50,298 INFO [collect_river_data] running in process 50503 2024-10-11 08:54:50,299 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:50,300 INFO [collect_river_data] publishing log messages to tcp://*:5566 2024-10-11 08:54:50,300 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:50,301 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-10-10 2024-10-11 08:54:50,303 INFO [collect_river_data] running in process 50505 2024-10-11 08:54:50,304 INFO [collect_river_data] running in process 50504 2024-10-11 08:54:50,304 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:50,305 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:50,306 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-10-11 08:54:50,307 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-10-11 08:54:50,307 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:50,308 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:50,309 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-10-10 2024-10-11 08:54:50,309 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-10-10 2024-10-11 08:54:50,348 INFO [download_live_ocean] downloaded 10204851 bytes from https://liveocean.apl.uw.edu/output/f2024.10.11/ubc.nc to /results/forcing/LiveOcean/downloaded/20241011/low_passed_UBC.nc 2024-10-11 08:54:50,349 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2024-10-11 08:54:50,375 INFO [make_turbidity_file] running in process 50506 2024-10-11 08:54:50,376 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:50,377 INFO [make_turbidity_file] publishing log messages to tcp://*:5569 2024-10-11 08:54:50,378 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:50,379 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-10-11 2024-10-11 08:54:50,480 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2024-10-10&endDT=2024-10-10 "HTTP/1.1 200 OK" 2024-10-11 08:54:50,484 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-10-10 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2024-10-11 08:54:50,484 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-10-10 completed 2024-10-11 08:54:50,486 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-11 08:54:50,591 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2024-10-10&endDT=2024-10-10 "HTTP/1.1 200 OK" 2024-10-11 08:54:50,595 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-10-10 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2024-10-11 08:54:50,596 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-10-10 completed 2024-10-11 08:54:50,598 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-11 08:54:50,633 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2024-10-10&endDT=2024-10-10 "HTTP/1.1 200 OK" 2024-10-11 08:54:50,637 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-10-10 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2024-10-11 08:54:50,637 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-10-10 completed 2024-10-11 08:54:50,640 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-11 08:54:50,644 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2024-10-10&endDT=2024-10-10 "HTTP/1.1 200 OK" 2024-10-11 08:54:50,647 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-10-10 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2024-10-11 08:54:50,648 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-10-10 completed 2024-10-11 08:54:50,649 INFO [crop_gribs] running in process 50511 2024-10-11 08:54:50,650 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:50,651 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-11 08:54:50,651 INFO [crop_gribs] publishing log messages to tcp://*:5570 2024-10-11 08:54:50,652 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:50,653 INFO [crop_gribs] cropping 2024-10-11 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2024-10-11 08:54:50,665 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241011/18/ 2024-10-11 08:54:50,697 INFO [collect_NeahBay_ssh] 2024-10-11T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2024-10-11 08:54:50,700 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-10-11 08:54:50,704 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 08:54:50,773 INFO [grib_to_netcdf] running in process 50534 2024-10-11 08:54:50,774 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:50,775 INFO [grib_to_netcdf] publishing log messages to tcp://*:5566 2024-10-11 08:54:50,776 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:50,777 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-10-11 nowcast and forecast runs 2024-10-11 08:54:51,378 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20241011/low_passed_UBC.nc 2024-10-11 08:54:51,378 INFO [download_live_ocean] 2024-10-11 Live Ocean file for Salish Sea western boundary downloaded 2024-10-11 08:54:51,380 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2024-10-11 08:54:51,523 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-10-11'], host='localhost') 2024-10-11 08:54:53,134 INFO [make_ssh_files] running in process 50684 2024-10-11 08:54:53,135 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:53,136 INFO [make_ssh_files] publishing log messages to tcp://*:5562 2024-10-11 08:54:53,137 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:53,138 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-10-11 NOAA Neah Bay 06Z observation and forecast values 2024-10-11 08:54:53,408 INFO [make_turbidity_file] 2024-10-11 Fraser River turbidity file creation complete 2024-10-11 08:54:53,410 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2024-10-11 08:54:53,495 INFO [make_live_ocean_files] running in process 50697 2024-10-11 08:54:53,496 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:54:53,497 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564 2024-10-11 08:54:53,498 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:54:53,500 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-10-11 Live Ocean run 2024-10-11 08:54:57,513 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d07.nc 2024-10-11 08:54:57,713 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d08.nc 2024-10-11 08:54:57,905 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d09.nc 2024-10-11 08:54:58,106 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d10.nc 2024-10-11 08:54:58,319 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d11.nc 2024-10-11 08:54:58,524 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d12.nc 2024-10-11 08:54:58,724 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d13.nc 2024-10-11 08:54:58,928 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d14.nc 2024-10-11 08:54:59,038 INFO [make_ssh_files] sea surface height boundary file for 2024-10-11 nowcast run created 2024-10-11 08:54:59,042 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-10-11 08:54:59,048 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-10-11 08:54:59,053 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-10-11 08:54:59,487 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m10d11.nc 2024-10-11 08:55:00,336 INFO [make_runoff_file] running in process 50753 2024-10-11 08:55:00,337 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:00,338 INFO [make_runoff_file] publishing log messages to tcp://*:5562 2024-10-11 08:55:00,339 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:00,368 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-10-11 08:55:00,439 INFO [make_runoff_file] 2024-10-11 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-10-11 08:55:00,441 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-10-11 08:55:01,012 INFO [make_v202111_runoff_file] running in process 50751 2024-10-11 08:55:01,014 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:01,014 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5562 2024-10-11 08:55:01,015 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:01,017 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-10-10 2024-10-11 08:55:03,546 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-10-10: /results/forcing/rivers/R202108Dailies_y2024m10d10.nc 2024-10-11 08:55:03,547 INFO [make_v202111_runoff_file] 2024-10-10 runoff file creation completed 2024-10-11 08:55:03,549 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-10-11 08:55:06,436 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m10d12.nc 2024-10-11 08:55:10,159 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m10d13.nc 2024-10-11 08:55:10,172 INFO [grib_to_netcdf] 2024-10-11 NEMO-atmos forcing file for nowcast+ created 2024-10-11 08:55:10,176 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-10-11 08:55:31,630 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m10d11.nc 2024-10-11 08:55:31,631 INFO [make_live_ocean_files] 2024-10-11 Live Ocean western boundary conditions files created 2024-10-11 08:55:31,633 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2024-10-11 08:55:31,641 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2024-10-11 08:55:31,645 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2024-10-11 08:55:31,649 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2024-10-11 08:55:31,653 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2024-10-11 08:55:31,658 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2024-10-11 08:55:32,838 INFO [ping_erddap] running in process 50797 2024-10-11 08:55:32,840 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:32,841 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-10-11 08:55:32,842 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:32,843 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2024-10-11 08:55:32,845 INFO [upload_forcing] running in process 50800 2024-10-11 08:55:32,846 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:32,847 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-11 08:55:32,848 INFO [upload_forcing] publishing log messages to tcp://*:5564 2024-10-11 08:55:32,849 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:32,867 INFO [upload_forcing] running in process 50801 2024-10-11 08:55:32,868 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:32,869 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-10-11 08:55:32,870 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:32,871 INFO [upload_forcing] running in process 50802 2024-10-11 08:55:32,872 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:32,873 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-10-11 08:55:32,873 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:32,902 INFO [upload_forcing] running in process 50799 2024-10-11 08:55:32,904 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:32,905 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-10-11 08:55:32,905 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:35,131 INFO [upload_forcing] nowcast+ 2024-10-11 forcing files upload to optimum-hindcast completed 2024-10-11 08:55:35,135 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-11 08:55:35,237 INFO [upload_forcing] nowcast+ 2024-10-11 forcing files upload to orcinus-nowcast-agrif completed 2024-10-11 08:55:35,241 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-11 08:55:42,196 INFO [upload_forcing] nowcast+ 2024-10-11 forcing files upload to robot.graham completed 2024-10-11 08:55:42,201 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-11 08:55:44,287 INFO [upload_forcing] nowcast+ 2024-10-11 forcing files upload to arbutus.cloud-nowcast completed 2024-10-11 08:55:44,291 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-11 08:55:44,301 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 08:55:45,501 INFO [make_forcing_links] running in process 50836 2024-10-11 08:55:45,502 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:45,504 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2024-10-11 08:55:45,505 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:46,625 INFO [make_forcing_links] nowcast+ 2024-10-11 forcing file links on arbutus.cloud-nowcast created 2024-10-11 08:55:46,629 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-11 08:55:46,635 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-10-11'], host='arbutus.cloud-nowcast') 2024-10-11 08:55:48,751 INFO [run_NEMO] running in process 14725 2024-10-11 08:55:48,753 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:48,754 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-10-11 08:55:48,762 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:49,622 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/11oct24nowcast_2024-10-11T085548.805164-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-10-11 08:55:49,639 INFO [run_NEMO] nowcast NEMO run for 2024-10-11 on arbutus.cloud-nowcast started 2024-10-11 08:55:49,644 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-10-11 08:55:49,654 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-10-11 08:55:51,686 INFO [watch_NEMO] running in process 14977 2024-10-11 08:55:51,687 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 08:55:51,688 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-10-11 08:55:51,689 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-10-11 08:55:51,759 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-10-11 09:00:51,761 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17975973 = 2024-10-11 05:01:20 UTC, 20.9% complete 2024-10-11 09:05:51,761 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17976459 = 2024-10-11 10:25:20 UTC, 43.4% complete 2024-10-11 09:10:51,762 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17976931 = 2024-10-11 15:40:00 UTC, 65.3% complete 2024-10-11 09:15:51,763 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17977449 = 2024-10-11 21:25:20 UTC, 89.3% complete 2024-10-11 09:20:51,768 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2024-10-11 09:20:51,771 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-10-11 09:20:51,929 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:20:51,935 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:20:53,201 INFO [make_forcing_links] running in process 50906 2024-10-11 09:20:53,202 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:20:53,203 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2024-10-11 09:20:53,204 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:20:53,204 INFO [download_results] running in process 50907 2024-10-11 09:20:53,205 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:20:53,206 INFO [download_results] publishing log messages to tcp://*:5564 2024-10-11 09:20:53,207 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:20:53,208 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/11oct24 to /results/SalishSea/nowcast-blue.202111 2024-10-11 09:20:54,476 INFO [make_forcing_links] ssh 2024-10-11 forcing file links on arbutus.cloud-nowcast created 2024-10-11 09:20:54,481 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-11 09:20:54,493 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-11'], host='arbutus.cloud-nowcast') 2024-10-11 09:21:01,645 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/11oct24forecast_2024-10-11T092100.120367-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-10-11 09:21:01,649 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-10-11 09:21:01,660 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-10-11 09:21:02,030 INFO [run_NEMO] forecast NEMO run for 2024-10-11 on arbutus.cloud-nowcast started 2024-10-11 09:21:13,493 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-10-11 09:26:13,502 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17978144 = 2024-10-12 05:08:40 UTC, 14.3% complete 2024-10-11 09:31:13,527 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17978619 = 2024-10-12 10:25:20 UTC, 29.0% complete 2024-10-11 09:36:13,535 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17979134 = 2024-10-12 16:08:40 UTC, 44.9% complete 2024-10-11 09:41:02,563 INFO [download_results] nowcast 2024-10-11 results files from arbutus.cloud-nowcast downloaded 2024-10-11 09:41:02,568 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-11 09:41:02,580 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:41:02,585 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-10-10'], host='localhost') 2024-10-11 09:41:02,590 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:41:04,591 INFO [make_CHS_currents_file] running in process 50999 2024-10-11 09:41:04,592 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:41:04,593 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2024-10-11 09:41:04,594 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:41:05,925 INFO [make_plots] running in process 50998 2024-10-11 09:41:05,927 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:41:05,928 INFO [make_plots] publishing log messages to tcp://*:5564 2024-10-11 09:41:05,929 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:41:05,957 INFO [make_plots] running in process 50996 2024-10-11 09:41:05,970 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:41:05,976 INFO [make_plots] publishing log messages to tcp://*:5565 2024-10-11 09:41:05,977 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:41:06,601 INFO [make_plots] preparing render list for 2024-10-11 NEMO nowcast-blue research figures 2024-10-11 09:41:06,602 INFO [make_plots] preparing render list for 2024-10-10 NEMO nowcast-blue comparison figures 2024-10-11 09:41:07,890 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-blue research render list 2024-10-11 09:41:08,572 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-blue research render list 2024-10-11 09:41:08,573 INFO [make_plots] added Currents_sections_and_surface figure to 2024-10-11 NEMO nowcast-blue research render list 2024-10-11 09:41:08,574 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-10-11 NEMO nowcast-blue research render list 2024-10-11 09:41:08,575 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-10-11 NEMO nowcast-blue research render list 2024-10-11 09:41:08,576 INFO [make_plots] starting to render nemo nowcast research 11oct24 figures 2024-10-11 09:41:12,860 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17979643 = 2024-10-12 21:48:00 UTC, 60.6% complete 2024-10-11 09:41:25,262 INFO [make_plots] added SH_wind figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,264 INFO [make_plots] added Compare_VENUS_East figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,265 INFO [make_plots] added Compare_VENUS_Central figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,266 INFO [make_plots] added Central_ADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,266 INFO [make_plots] added Central_depavADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,267 INFO [make_plots] added Central_timeavADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,268 INFO [make_plots] added East_ADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,269 INFO [make_plots] added East_depavADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,269 INFO [make_plots] added East_timeavADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,270 INFO [make_plots] added ddl_ADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,271 INFO [make_plots] added ddl_depavADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,271 INFO [make_plots] added ddl_timeavADCP figure to 2024-10-10 NEMO nowcast-blue comparison render list 2024-10-11 09:41:25,272 INFO [make_plots] starting to render nemo nowcast comparison 10oct24 figures 2024-10-11 09:41:36,036 INFO [make_CHS_currents_file] Made CHS currents file for 2024-10-11 for nowcast 2024-10-11 09:41:36,038 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-10-11 09:42:18,592 INFO [make_plots] finished rendering nemo nowcast research 11oct24 figures 2024-10-11 09:42:18,594 INFO [make_plots] nemo research plots for 2024-10-11 nowcast completed 2024-10-11 09:42:18,612 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-11 09:42:48,651 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available 2024-10-11 09:42:49,190 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-10-11 09:42:49,686 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-10-11 09:42:50,177 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-10-11 09:42:50,670 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-10-11 09:42:51,160 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-10-11 09:42:51,651 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-10-11 09:42:52,145 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-10-11 09:42:52,640 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-10-11 09:42:53,129 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-10-11 09:42:53,131 INFO [make_plots] finished rendering nemo nowcast comparison 10oct24 figures 2024-10-11 09:42:53,207 INFO [make_plots] nemo comparison plots for 2024-10-10 nowcast completed 2024-10-11 09:42:53,210 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-11 09:46:12,861 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17980143 = 2024-10-13 03:21:20 UTC, 76.0% complete 2024-10-11 09:51:12,863 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17980665 = 2024-10-13 09:09:20 UTC, 92.1% complete 2024-10-11 09:56:12,867 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2024-10-11 09:56:12,871 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-10-11 09:56:13,046 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-11'], host='arbutus.cloud-nowcast') 2024-10-11 09:56:13,167 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-11'], host='arbutus.cloud-nowcast') 2024-10-11 09:56:13,172 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:56:13,178 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:56:13,182 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:56:13,187 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:56:13,191 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:56:14,752 INFO [upload_forcing] running in process 51241 2024-10-11 09:56:14,753 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:14,754 INFO [upload_forcing] publishing log messages to tcp://*:5562 2024-10-11 09:56:14,755 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:14,756 INFO [download_results] running in process 51242 2024-10-11 09:56:14,757 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:14,758 INFO [download_results] publishing log messages to tcp://*:5564 2024-10-11 09:56:14,759 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:14,760 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/11oct24 to /results/SalishSea/forecast.202111 2024-10-11 09:56:14,762 INFO [upload_forcing] running in process 51239 2024-10-11 09:56:14,763 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:14,764 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-10-11 09:56:14,765 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:14,776 INFO [upload_forcing] running in process 51240 2024-10-11 09:56:14,777 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:14,778 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-10-11 09:56:14,779 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:14,784 INFO [upload_forcing] running in process 51238 2024-10-11 09:56:14,785 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:14,787 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-10-11 09:56:14,788 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:15,411 INFO [upload_forcing] turbidity 2024-10-11 forcing files upload to optimum-hindcast completed 2024-10-11 09:56:15,416 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-11 09:56:15,437 INFO [upload_forcing] turbidity 2024-10-11 forcing files upload to orcinus-nowcast-agrif completed 2024-10-11 09:56:15,441 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-11 09:56:15,454 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:56:16,924 INFO [make_forcing_links] running in process 51274 2024-10-11 09:56:16,925 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:16,926 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2024-10-11 09:56:16,928 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:17,678 INFO [make_forcing_links] nowcast-agrif 2024-10-11 forcing file links on orcinus-nowcast-agrif created 2024-10-11 09:56:17,683 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-11 09:56:17,701 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:56:18,550 INFO [upload_forcing] turbidity 2024-10-11 forcing files upload to robot.graham completed 2024-10-11 09:56:18,554 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-11 09:56:18,899 INFO [run_NEMO_agrif] running in process 51283 2024-10-11 09:56:18,901 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:18,902 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5562 2024-10-11 09:56:18,903 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:23,772 INFO [upload_forcing] turbidity 2024-10-11 forcing files upload to arbutus.cloud-nowcast completed 2024-10-11 09:56:23,776 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-11 09:56:23,794 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 09:56:25,010 INFO [make_forcing_links] running in process 51299 2024-10-11 09:56:25,011 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:25,012 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-10-11 09:56:25,013 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:28,252 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-10-11 forecast run 2024-10-11 09:56:28,257 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-10-11 09:56:35,674 INFO [run_NEMO_agrif] job id for 11oct24nowcast-agrif: 11410234.orca2.ibb 2024-10-11 09:56:35,675 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-10-11 queued on orcinus-nowcast-agrif 2024-10-11 09:56:35,680 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker 2024-10-11 09:56:35,695 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11410234.orca2.ibb'], host='localhost') 2024-10-11 09:56:36,850 INFO [watch_NEMO_agrif] running in process 51306 2024-10-11 09:56:36,851 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 09:56:36,852 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5562 2024-10-11 09:56:36,853 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-10-11 09:56:51,922 ERROR [watch_NEMO_agrif] Exception (client): Error reading SSH protocol banner 2024-10-11 09:56:52,385 ERROR [watch_NEMO_agrif] Traceback (most recent call last): 2024-10-11 09:56:52,386 ERROR [watch_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2327, in _check_banner 2024-10-11 09:56:52,388 ERROR [watch_NEMO_agrif] buf = self.packetizer.readline(timeout) 2024-10-11 09:56:52,390 ERROR [watch_NEMO_agrif] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-11 09:56:52,392 ERROR [watch_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 381, in readline 2024-10-11 09:56:52,394 ERROR [watch_NEMO_agrif] buf += self._read_timeout(timeout) 2024-10-11 09:56:52,396 ERROR [watch_NEMO_agrif] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-11 09:56:52,398 ERROR [watch_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 626, in _read_timeout 2024-10-11 09:56:52,400 ERROR [watch_NEMO_agrif] raise socket.timeout() 2024-10-11 09:56:52,401 ERROR [watch_NEMO_agrif] TimeoutError 2024-10-11 09:56:52,403 ERROR [watch_NEMO_agrif] 2024-10-11 09:56:52,404 ERROR [watch_NEMO_agrif] During handling of the above exception, another exception occurred: 2024-10-11 09:56:52,406 ERROR [watch_NEMO_agrif] 2024-10-11 09:56:52,408 ERROR [watch_NEMO_agrif] Traceback (most recent call last): 2024-10-11 09:56:52,410 ERROR [watch_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2143, in run 2024-10-11 09:56:52,412 ERROR [watch_NEMO_agrif] self._check_banner() 2024-10-11 09:56:52,413 ERROR [watch_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2331, in _check_banner 2024-10-11 09:56:52,415 ERROR [watch_NEMO_agrif] raise SSHException( 2024-10-11 09:56:52,417 ERROR [watch_NEMO_agrif] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-10-11 09:56:52,419 ERROR [watch_NEMO_agrif] 2024-10-11 09:56:53,008 INFO [watch_NEMO_agrif] watching 11oct24nowcast-agrif job 11410234 on orcinus-nowcast-agrif 2024-10-11 09:56:53,243 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif job 11410234 is queued on orcinus-nowcast-agrif 2024-10-11 09:57:06,583 INFO [make_forcing_links] nowcast-green 2024-10-11 forcing file links on arbutus.cloud-nowcast created 2024-10-11 09:57:06,588 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-11 09:57:06,605 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-10-11'], host='arbutus.cloud-nowcast') 2024-10-11 09:57:14,340 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-10-11 forecast run 2024-10-11 09:57:14,342 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-10-11 09:57:14,360 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-10-11'], host='arbutus.cloud-nowcast') 2024-10-11 09:57:18,126 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-10-11 09:57:18,142 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2024-10-11 09:57:25,498 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-10-11 09:57:25,518 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-10-11 09:57:30,088 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-10-11 09:57:37,383 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-11 09:57:53,452 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif job 11410234 is queued on orcinus-nowcast-agrif 2024-10-11 09:58:54,501 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2024-10-11 10:02:30,113 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17975682 = 2024-10-11 01:47:20 UTC, 7.5% complete 2024-10-11 10:02:37,392 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete 2024-10-11 10:03:55,038 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7996537 = 2024-10-11 02:24:00 UTC, 10.0% complete 2024-10-11 10:07:30,106 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17975856 = 2024-10-11 03:43:20 UTC, 15.5% complete 2024-10-11 10:07:37,404 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 183, 42.4% complete 2024-10-11 10:08:55,514 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7996795 = 2024-10-11 05:16:00 UTC, 22.0% complete 2024-10-11 10:12:30,085 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17976032 = 2024-10-11 05:40:40 UTC, 23.7% complete 2024-10-11 10:12:37,377 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 270, 62.5% complete 2024-10-11 10:13:55,995 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7997055 = 2024-10-11 08:09:20 UTC, 34.0% complete 2024-10-11 10:17:30,128 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17976207 = 2024-10-11 07:37:20 UTC, 31.8% complete 2024-10-11 10:17:37,406 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 354, 81.9% complete 2024-10-11 10:18:56,479 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7997316 = 2024-10-11 11:03:20 UTC, 46.1% complete 2024-10-11 10:22:30,103 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17976383 = 2024-10-11 09:34:40 UTC, 39.9% complete 2024-10-11 10:22:37,420 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-11 10:23:56,958 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7997579 = 2024-10-11 13:58:40 UTC, 58.3% complete 2024-10-11 10:25:31,040 INFO [download_results] forecast 2024-10-11 results files from arbutus.cloud-nowcast downloaded 2024-10-11 10:25:31,045 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-11 10:25:31,064 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 10:25:33,498 INFO [make_CHS_currents_file] running in process 51379 2024-10-11 10:25:33,500 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:25:33,501 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564 2024-10-11 10:25:33,502 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:26:14,467 INFO [make_CHS_currents_file] Made CHS currents file for 2024-10-11 for forecast 2024-10-11 10:26:14,471 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-10-11 10:26:14,492 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 10:26:15,531 INFO [update_forecast_datasets] running in process 51402 2024-10-11 10:26:15,532 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:26:15,533 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2024-10-11 10:26:15,534 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:26:15,535 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/11oct24 run 2024-10-11 10:26:15,789 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/11oct24 run: /results/SalishSea/rolling-forecasts/nemo 2024-10-11 10:26:15,790 INFO [update_forecast_datasets] nemo 2024-10-11 forecast rolling forecast datasets updated 2024-10-11 10:26:15,792 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-10-11 10:26:15,806 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-10-11 10:26:15,810 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 10:26:15,814 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 10:26:16,897 INFO [ping_erddap] running in process 51407 2024-10-11 10:26:16,899 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:26:16,900 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-10-11 10:26:16,901 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:26:16,909 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-10-11 10:26:16,911 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-11 10:26:17,972 INFO [make_surface_current_tiles] running in process 51410 2024-10-11 10:26:17,973 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:26:17,974 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564 2024-10-11 10:26:17,975 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:26:17,977 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-10-11 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/11oct24 2024-10-11 10:26:19,856 INFO [make_plots] running in process 51409 2024-10-11 10:26:19,857 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:26:19,858 INFO [make_plots] publishing log messages to tcp://*:5565 2024-10-11 10:26:19,859 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:26:21,175 INFO [make_plots] preparing render list for 2024-10-11 NEMO forecast publish figures 2024-10-11 10:26:21,298 INFO [make_plots] added Website_thumbnail figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,299 INFO [make_plots] added Threshold_website figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,300 INFO [make_plots] added PA_tidal_predictions figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,301 INFO [make_plots] added Boundary Bay figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,302 INFO [make_plots] added Campbell River figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,303 INFO [make_plots] added Cherry Point figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,304 INFO [make_plots] added Friday Harbor figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,304 INFO [make_plots] added Halfmoon Bay figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,305 INFO [make_plots] added Nanaimo figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,306 INFO [make_plots] added Neah Bay figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,307 INFO [make_plots] added New Westminster figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,308 INFO [make_plots] added Patricia Bay figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,309 INFO [make_plots] added Point Atkinson figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,309 INFO [make_plots] added Port Renfrew figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,310 INFO [make_plots] added Sand Heads figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,311 INFO [make_plots] added Sandy Cove figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,312 INFO [make_plots] added Squamish figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,313 INFO [make_plots] added Victoria figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,314 INFO [make_plots] added Woodwards Landing figure to 2024-10-11 NEMO forecast publish render list 2024-10-11 10:26:21,315 INFO [make_plots] starting to render nemo forecast publish 11oct24 figures 2024-10-11 10:27:29,435 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17976558 = 2024-10-11 11:31:20 UTC, 48.0% complete 2024-10-11 10:27:36,730 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-10-11 10:27:36,734 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-10-11 10:27:36,910 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 10:27:36,915 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-11'], host='arbutus.cloud-nowcast') 2024-10-11 10:27:38,547 INFO [download_wwatch3_results] running in process 51741 2024-10-11 10:27:38,547 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:27:38,548 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566 2024-10-11 10:27:38,549 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:27:38,921 INFO [run_ww3] running in process 17547 2024-10-11 10:27:38,922 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:27:38,923 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-10-11 10:27:38,924 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:27:38,927 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-10-11T102738.924648-0700 2024-10-11 10:27:38,928 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-10-11T102738.924648-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-10-11 10:27:38,953 INFO [run_ww3] forecast WaveWatch3 run for 2024-10-11 on arbutus.cloud-nowcast started 2024-10-11 10:27:38,967 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-10-11 10:27:38,969 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-10-11 10:27:40,958 INFO [watch_ww3] running in process 17701 2024-10-11 10:27:40,959 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:27:40,960 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-10-11 10:27:40,961 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:27:40,994 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-11 10:28:57,437 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7997839 = 2024-10-11 16:52:00 UTC, 70.3% complete 2024-10-11 10:29:37,634 INFO [download_wwatch3_results] nowcast 2024-10-11 results files from arbutus.cloud-nowcast downloaded 2024-10-11 10:29:37,637 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-10-11 10:32:29,436 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17976733 = 2024-10-11 13:28:00 UTC, 56.1% complete 2024-10-11 10:32:40,995 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 81, 12.5% complete 2024-10-11 10:33:57,879 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7998099 = 2024-10-11 19:45:20 UTC, 82.4% complete 2024-10-11 10:36:57,549 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/11oct24 2024-10-11 10:37:29,437 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17976909 = 2024-10-11 15:25:20 UTC, 64.3% complete 2024-10-11 10:37:40,996 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 165, 25.5% complete 2024-10-11 10:38:58,353 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7998358 = 2024-10-11 22:38:00 UTC, 94.3% complete 2024-10-11 10:39:37,129 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-10-11 10:39:37,130 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-10-11 10:40:12,210 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:41:36,216 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/11oct24 2024-10-11 10:41:36,228 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-10-11 forecast into /results/nowcast-sys/figures/surface_currents/forecast/11oct24 2024-10-11 10:41:36,229 INFO [make_surface_current_tiles] surface current tile figures for 2024-10-11 forecast completed 2024-10-11 10:41:36,551 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-10-11 10:41:58,635 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:41:58,825 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-10-11 10:41:59,496 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:42:29,440 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17977084 = 2024-10-11 17:22:00 UTC, 72.4% complete 2024-10-11 10:42:40,999 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 252, 38.9% complete 2024-10-11 10:43:08,176 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:43:43,473 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:43:58,780 INFO [watch_NEMO_agrif] 11oct24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7998480 = 2024-10-11 23:59:20 UTC, 100.0% complete 2024-10-11 10:44:18,650 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:44:54,000 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:45:28,413 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:46:03,800 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:46:38,679 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:46:38,848 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-10-11 10:46:39,081 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:47:14,703 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-10-06 00:00:00Z to 2024-10-13 11:55:00Z 2024-10-11 10:47:29,443 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17977259 = 2024-10-11 19:18:40 UTC, 80.5% complete 2024-10-11 10:47:41,002 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 336, 51.9% complete 2024-10-11 10:47:49,851 INFO [make_plots] finished rendering nemo forecast publish 11oct24 figures 2024-10-11 10:47:49,852 INFO [make_plots] nemo publish plots for 2024-10-11 forecast completed 2024-10-11 10:47:49,860 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-11 10:47:50,026 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 10:47:52,436 INFO [make_feeds] running in process 53586 2024-10-11 10:47:52,437 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:47:52,439 INFO [make_feeds] publishing log messages to tcp://*:5564 2024-10-11 10:47:52,440 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:48:21,187 INFO [make_feeds] ATOM feeds for 2024-10-11 forecast run completed 2024-10-11 10:48:21,189 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-10-11 10:48:59,002 INFO [watch_NEMO_agrif] NEMO AGRIF run 11410234.orca2.ibb on orcinus-nowcast-agrif completed 2024-10-11 10:48:59,007 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker 2024-10-11 10:48:59,331 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 10:49:00,588 INFO [download_results] running in process 53611 2024-10-11 10:49:00,589 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:49:00,590 INFO [download_results] publishing log messages to tcp://*:5562 2024-10-11 10:49:00,591 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:49:00,592 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/11oct24 to /results/SalishSea/nowcast-agrif.201702 2024-10-11 10:49:22,080 INFO [download_results] nowcast-agrif 2024-10-11 results files from orcinus-nowcast-agrif downloaded 2024-10-11 10:49:22,086 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-11 10:49:22,256 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 10:49:25,662 INFO [make_plots] running in process 53617 2024-10-11 10:49:25,663 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 10:49:25,664 INFO [make_plots] publishing log messages to tcp://*:5562 2024-10-11 10:49:25,665 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-11 10:49:26,814 INFO [make_plots] preparing render list for 2024-10-11 NEMO nowcast-agrif research figures 2024-10-11 10:49:26,816 INFO [make_plots] added baynes_sound_surface figure to 2024-10-11 NEMO nowcast-agrif research render list 2024-10-11 10:49:26,816 INFO [make_plots] starting to render nemo nowcast-agrif research 11oct24 figures 2024-10-11 10:49:37,509 INFO [make_plots] finished rendering nemo nowcast-agrif research 11oct24 figures 2024-10-11 10:49:37,512 INFO [make_plots] nemo research plots for 2024-10-11 nowcast-agrif completed 2024-10-11 10:49:37,516 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-11 10:52:29,446 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17977434 = 2024-10-11 21:15:20 UTC, 88.6% complete 2024-10-11 10:52:41,005 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 426, 65.7% complete 2024-10-11 10:57:29,448 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17977609 = 2024-10-11 23:12:00 UTC, 96.7% complete 2024-10-11 10:57:41,006 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 507, 78.2% complete 2024-10-11 11:02:29,450 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17977680 = 2024-10-11 23:59:20 UTC, 100.0% complete 2024-10-11 11:02:41,008 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 597, 92.1% complete 2024-10-11 11:07:29,458 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2024-10-11 11:07:29,462 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-10-11 11:07:29,777 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 11:07:31,228 INFO [download_results] running in process 53699 2024-10-11 11:07:31,229 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:07:31,231 INFO [download_results] publishing log messages to tcp://*:5562 2024-10-11 11:07:31,232 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:07:31,233 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/11oct24 to /results2/SalishSea/nowcast-green.202111 2024-10-11 11:07:41,650 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-11 11:12:41,444 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-10-11 11:12:41,447 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-10-11 11:12:41,775 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 11:12:43,079 INFO [download_wwatch3_results] running in process 53721 2024-10-11 11:12:43,082 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:12:43,083 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564 2024-10-11 11:12:43,084 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:18:51,463 INFO [download_wwatch3_results] forecast 2024-10-11 results files from arbutus.cloud-nowcast downloaded 2024-10-11 11:18:51,467 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-10-11 11:18:51,640 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 11:18:52,957 INFO [update_forecast_datasets] running in process 53743 2024-10-11 11:18:52,958 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:18:52,959 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2024-10-11 11:18:52,960 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:18:52,961 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/11oct24 run 2024-10-11 11:18:52,963 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/11oct24 run 2024-10-11 11:18:53,137 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/11oct24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-10-11 11:18:53,137 INFO [update_forecast_datasets] wwatch3 2024-10-11 forecast rolling forecast datasets updated 2024-10-11 11:18:53,140 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-10-11 11:18:53,314 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-10-11 11:18:54,507 INFO [ping_erddap] running in process 53748 2024-10-11 11:18:54,509 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:18:54,510 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-10-11 11:18:54,511 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:18:54,513 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-10-11 11:18:54,515 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-11 11:18:54,692 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 11:18:58,304 INFO [make_plots] running in process 53753 2024-10-11 11:18:58,305 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:18:58,306 INFO [make_plots] publishing log messages to tcp://*:5564 2024-10-11 11:18:58,308 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:18:58,309 INFO [make_plots] preparing render list for 2024-10-11 WaveWatch3 forecast publish figures 2024-10-11 11:18:58,310 INFO [make_plots] added Halibut Bank figure to 2024-10-11 WaveWatch3 forecast publish render list 2024-10-11 11:18:58,311 INFO [make_plots] added Sentry Shoal figure to 2024-10-11 WaveWatch3 forecast publish render list 2024-10-11 11:18:58,312 INFO [make_plots] starting to render wwatch3 forecast publish 11oct24 figures 2024-10-11 11:18:59,257 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-10-11 11:19:06,847 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-10-11 11:19:13,250 INFO [make_plots] finished rendering wwatch3 forecast publish 11oct24 figures 2024-10-11 11:19:13,250 INFO [make_plots] wwatch3 publish plots for 2024-10-11 forecast completed 2024-10-11 11:19:13,254 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-11 11:43:31,909 INFO [download_results] nowcast-green 2024-10-11 results files from arbutus.cloud-nowcast downloaded 2024-10-11 11:43:31,915 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-11 11:43:32,087 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 11:43:32,092 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2024-10-11 11:43:32,096 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 11:43:32,103 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 11:43:32,108 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-10-11'], host='localhost') 2024-10-11 11:43:33,463 INFO [ping_erddap] running in process 53846 2024-10-11 11:43:33,463 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:43:33,464 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-10-11 11:43:33,465 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:43:33,470 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2024-10-11 11:43:33,471 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-11 11:43:34,570 INFO [make_averaged_dataset] running in process 53849 2024-10-11 11:43:34,572 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:43:34,573 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562 2024-10-11 11:43:34,574 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:43:34,574 INFO [make_averaged_dataset] creating day-averaged dataset for 11-Oct-2024 physics 2024-10-11 11:43:34,580 INFO [make_averaged_dataset] running in process 53847 2024-10-11 11:43:34,581 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:43:34,582 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564 2024-10-11 11:43:34,583 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:43:34,584 INFO [make_averaged_dataset] creating day-averaged dataset for 11-Oct-2024 biology 2024-10-11 11:43:34,597 INFO [make_averaged_dataset] running in process 53848 2024-10-11 11:43:34,598 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:43:34,599 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565 2024-10-11 11:43:34,600 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:43:34,601 INFO [make_averaged_dataset] creating day-averaged dataset for 11-Oct-2024 chemistry 2024-10-11 11:43:35,704 INFO [make_plots] running in process 53844 2024-10-11 11:43:35,706 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 11:43:35,706 INFO [make_plots] publishing log messages to tcp://*:5566 2024-10-11 11:43:35,708 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-11 11:43:36,468 INFO [make_plots] preparing render list for 2024-10-11 NEMO nowcast-green research figures 2024-10-11 11:43:38,180 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:38,873 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:39,585 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:40,282 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:41,006 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:41,712 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:42,416 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:43,228 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:44,042 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:44,863 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:45,687 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:46,397 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:47,095 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:47,096 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:47,096 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:47,097 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-10-11 NEMO nowcast-green research render list 2024-10-11 11:43:47,098 INFO [make_plots] starting to render nemo nowcast-green research 11oct24 figures 2024-10-11 11:43:51,917 INFO [make_averaged_dataset] day-averaged dataset for 11-Oct-2024 physics created 2024-10-11 11:43:51,920 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-10-11 11:44:26,494 INFO [make_averaged_dataset] day-averaged dataset for 11-Oct-2024 chemistry created 2024-10-11 11:44:26,498 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-10-11 11:44:42,260 INFO [make_averaged_dataset] day-averaged dataset for 11-Oct-2024 biology created 2024-10-11 11:44:42,264 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-10-11 12:51:42,137 INFO [make_plots] finished rendering nemo nowcast-green research 11oct24 figures 2024-10-11 12:51:42,305 INFO [make_plots] nemo research plots for 2024-10-11 nowcast-green completed 2024-10-11 12:51:42,373 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-11 14:54:49,383 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20241011/18/ 2024-10-11 14:54:49,384 INFO [collect_weather] 2024-10-11 2.5km weather forecast 18 collection complete 2024-10-11 14:54:49,387 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-10-11 14:54:49,963 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2024-10-11 14:54:49,971 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2024-10-11 14:54:49,976 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2024-10-11 14:54:49,981 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-10-12'], host='localhost') 2024-10-11 14:54:50,663 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241011/18/ 2024-10-11 14:54:50,664 INFO [crop_gribs] 2024-10-11 18 GRIBs cropping complete 2024-10-11 14:54:50,666 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-10-11 14:54:51,513 INFO [collect_weather] running in process 54857 2024-10-11 14:54:51,514 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 14:54:51,515 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-10-11 14:54:51,517 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-11 14:54:51,517 INFO [download_weather] running in process 54854 2024-10-11 14:54:51,518 INFO [download_weather] running in process 54856 2024-10-11 14:54:51,519 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 14:54:51,520 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 14:54:51,521 INFO [download_weather] publishing log messages to tcp://*:5563 2024-10-11 14:54:51,522 INFO [download_weather] publishing log messages to tcp://*:5564 2024-10-11 14:54:51,523 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-11 14:54:51,524 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-11 14:54:51,525 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20241011 2024-10-11 14:54:51,525 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20241011 2024-10-11 14:54:51,558 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2024-10-11 14:54:52,679 INFO [crop_gribs] running in process 54858 2024-10-11 14:54:52,680 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 14:54:52,681 INFO [crop_gribs] publishing log messages to tcp://*:5565 2024-10-11 14:54:52,682 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-10-11 14:54:52,683 INFO [crop_gribs] cropping 2024-10-12 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2024-10-11 14:54:52,697 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241012/00/ 2024-10-11 14:58:31,709 INFO [download_weather] 2024-10-11 1km weather forecast 00 downloads complete 2024-10-11 14:58:31,712 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-10-11 14:58:33,624 INFO [download_weather] 2024-10-11 1km weather forecast 12 downloads complete 2024-10-11 14:58:33,627 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-10-11 20:56:26,299 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20241012/00/ 2024-10-11 20:56:26,300 INFO [collect_weather] 2024-10-12 2.5km weather forecast 00 collection complete 2024-10-11 20:56:26,303 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-10-11 20:56:26,644 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2024-10-11 20:56:26,651 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-10-12'], host='localhost') 2024-10-11 20:56:27,427 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241012/00/ 2024-10-11 20:56:27,428 INFO [crop_gribs] 2024-10-12 00 GRIBs cropping complete 2024-10-11 20:56:27,431 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-10-11 20:56:27,864 INFO [collect_weather] running in process 56932 2024-10-11 20:56:27,865 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 20:56:27,866 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-10-11 20:56:27,867 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-11 20:56:27,882 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2024-10-11 20:56:28,658 INFO [crop_gribs] running in process 56934 2024-10-11 20:56:28,659 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-11 20:56:28,660 INFO [crop_gribs] publishing log messages to tcp://*:5563 2024-10-11 20:56:28,661 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-10-11 20:56:28,661 INFO [crop_gribs] cropping 2024-10-12 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2024-10-11 20:56:28,676 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241012/06/ 2024-10-12 02:53:52,595 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20241012/06/ 2024-10-12 02:53:52,597 INFO [collect_weather] 2024-10-12 2.5km weather forecast 06 collection complete 2024-10-12 02:53:52,599 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-10-12 02:53:52,945 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,951 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,956 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,959 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,964 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,970 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,974 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,977 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,981 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,986 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:52,997 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:53,010 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:53,027 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:53,041 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:53,050 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2024-10-12 02:53:53,056 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2024-10-12 02:53:53,065 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2024-10-12 02:53:53,082 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-10-11'], host='localhost') 2024-10-12 02:53:53,094 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2024-10-12 02:53:53,108 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2024-10-12 02:53:53,135 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2024-10-12 02:53:54,172 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241012/06/ 2024-10-12 02:53:54,173 INFO [crop_gribs] 2024-10-12 06 GRIBs cropping complete 2024-10-12 02:53:54,174 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-10-12 02:53:55,128 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2024-10-12 02:53:56,077 INFO [collect_weather] running in process 58262 2024-10-12 02:53:56,078 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:56,079 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-10-12 02:53:56,080 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:56,103 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2024-10-12 02:53:56,183 INFO [collect_NeahBay_ssh] running in process 58261 2024-10-12 02:53:56,184 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:56,185 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563 2024-10-12 02:53:56,185 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:56,186 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-10-12 2024-10-12 02:53:57,060 INFO [collect_river_data] running in process 58245 2024-10-12 02:53:57,061 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,062 INFO [collect_river_data] publishing log messages to tcp://*:5564 2024-10-12 02:53:57,063 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,065 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-10-11 2024-10-12 02:53:57,103 INFO [collect_river_data] running in process 58241 2024-10-12 02:53:57,104 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,104 INFO [collect_river_data] publishing log messages to tcp://*:5565 2024-10-12 02:53:57,105 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,106 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-10-11 2024-10-12 02:53:57,114 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2024-10-12 02:53:57,115 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-10-11 completed 2024-10-12 02:53:57,118 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:57,135 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2024-10-12 02:53:57,135 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-10-11 completed 2024-10-12 02:53:57,143 INFO [collect_river_data] running in process 58254 2024-10-12 02:53:57,144 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,148 INFO [collect_river_data] publishing log messages to tcp://*:5566 2024-10-12 02:53:57,149 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,149 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-10-11 2024-10-12 02:53:57,188 INFO [collect_river_data] running in process 58244 2024-10-12 02:53:57,190 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,190 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-10-12 02:53:57,191 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,192 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-10-11 2024-10-12 02:53:57,194 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2024-10-12 02:53:57,194 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-10-11 completed 2024-10-12 02:53:57,195 INFO [collect_river_data] running in process 58243 2024-10-12 02:53:57,196 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,196 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-10-12 02:53:57,197 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,198 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-10-11 2024-10-12 02:53:57,199 INFO [collect_river_data] running in process 58246 2024-10-12 02:53:57,200 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,201 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-10-12 02:53:57,202 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,202 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-10-11 2024-10-12 02:53:57,206 INFO [collect_river_data] running in process 58252 2024-10-12 02:53:57,207 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,209 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-10-12 02:53:57,209 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,210 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-10-11 2024-10-12 02:53:57,228 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-10-11 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2024-10-12 02:53:57,229 INFO [collect_river_data] ECCC Fraser river data collection for 2024-10-11 completed 2024-10-12 02:53:57,234 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-10-11 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2024-10-12 02:53:57,235 INFO [collect_river_data] ECCC Englishman river data collection for 2024-10-11 completed 2024-10-12 02:53:57,237 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2024-10-12 02:53:57,238 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-10-11 completed 2024-10-12 02:53:57,245 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2024-10-12 02:53:57,245 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-10-11 completed 2024-10-12 02:53:57,257 INFO [collect_river_data] running in process 58242 2024-10-12 02:53:57,258 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,258 INFO [collect_river_data] publishing log messages to tcp://*:5571 2024-10-12 02:53:57,259 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,260 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-10-11 2024-10-12 02:53:57,264 INFO [collect_river_data] running in process 58249 2024-10-12 02:53:57,265 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,266 INFO [collect_river_data] publishing log messages to tcp://*:5572 2024-10-12 02:53:57,267 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,268 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-10-11 2024-10-12 02:53:57,270 INFO [collect_river_data] running in process 58239 2024-10-12 02:53:57,271 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,272 INFO [collect_river_data] publishing log messages to tcp://*:5573 2024-10-12 02:53:57,272 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,273 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-10-11 2024-10-12 02:53:57,277 INFO [collect_river_data] running in process 58247 2024-10-12 02:53:57,278 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,280 INFO [collect_river_data] publishing log messages to tcp://*:5574 2024-10-12 02:53:57,280 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,281 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-10-11 2024-10-12 02:53:57,294 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2024-10-12 02:53:57,295 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-10-11 completed 2024-10-12 02:53:57,296 INFO [collect_river_data] running in process 58251 2024-10-12 02:53:57,297 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,298 INFO [collect_river_data] publishing log messages to tcp://*:5575 2024-10-12 02:53:57,299 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,299 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-10-11 2024-10-12 02:53:57,301 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2024-10-12 02:53:57,302 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-10-11 completed 2024-10-12 02:53:57,307 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-10-11 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2024-10-12 02:53:57,308 INFO [collect_river_data] ECCC Capilano river data collection for 2024-10-11 completed 2024-10-12 02:53:57,323 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2024-10-12 02:53:57,324 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-10-11 completed 2024-10-12 02:53:57,328 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/RobertsCreek_flow 2024-10-12 02:53:57,329 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-10-11 completed 2024-10-12 02:53:57,337 INFO [collect_river_data] running in process 58253 2024-10-12 02:53:57,338 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,339 INFO [collect_river_data] publishing log messages to tcp://*:5576 2024-10-12 02:53:57,340 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,340 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-10-11 2024-10-12 02:53:57,348 INFO [collect_river_data] running in process 58250 2024-10-12 02:53:57,350 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,351 INFO [collect_river_data] publishing log messages to tcp://*:5577 2024-10-12 02:53:57,352 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,352 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-10-11 2024-10-12 02:53:57,377 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2024-10-12 02:53:57,377 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-10-11 completed 2024-10-12 02:53:57,386 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-10-11 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2024-10-12 02:53:57,387 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-10-11 completed 2024-10-12 02:53:57,437 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:57,517 INFO [collect_NeahBay_ssh] 2024-10-12T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2024-10-12 02:53:57,624 INFO [get_vfpa_hadcp] running in process 58260 2024-10-12 02:53:57,625 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,626 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578 2024-10-12 02:53:57,627 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,627 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-10-11 2024-10-12 02:53:57,629 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:57,688 INFO [crop_gribs] running in process 58263 2024-10-12 02:53:57,689 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,690 INFO [crop_gribs] publishing log messages to tcp://*:5564 2024-10-12 02:53:57,690 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,691 INFO [crop_gribs] cropping 2024-10-12 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2024-10-12 02:53:57,704 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241012/12/ 2024-10-12 02:53:57,862 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:57,864 INFO [get_onc_ferry] running in process 58258 2024-10-12 02:53:57,865 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,866 INFO [get_onc_ferry] publishing log messages to tcp://*:5579 2024-10-12 02:53:57,867 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,867 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-10-11 2024-10-12 02:53:57,886 INFO [get_onc_ctd] running in process 58256 2024-10-12 02:53:57,887 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,888 INFO [get_onc_ctd] publishing log messages to tcp://*:5565 2024-10-12 02:53:57,889 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,890 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-10-11 2024-10-12 02:53:57,890 INFO [get_onc_ctd] running in process 58257 2024-10-12 02:53:57,891 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:57,892 INFO [get_onc_ctd] publishing log messages to tcp://*:5590 2024-10-12 02:53:57,893 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:57,894 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-10-11 2024-10-12 02:53:58,047 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:58,181 INFO [grib_to_netcdf] running in process 58524 2024-10-12 02:53:58,183 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:53:58,183 INFO [grib_to_netcdf] publishing log messages to tcp://*:5566 2024-10-12 02:53:58,184 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:53:58,185 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-10-12 forecast2 run 2024-10-12 02:53:58,224 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:58,403 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:58,579 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:58,764 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:58,950 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:59,131 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:59,333 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-10-11 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202410.nc 2024-10-12 02:53:59,334 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-10 netcdf file 2024-10-12 02:53:59,368 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:59,560 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:59,731 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-10-12 02:53:59,899 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-10-12 02:54:00,067 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-10-12'], host='localhost') 2024-10-12 02:54:00,072 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2024-10-12 02:54:00,297 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2024-10-12 02:54:01,415 INFO [ping_erddap] running in process 58726 2024-10-12 02:54:01,416 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:01,417 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-12 02:54:01,418 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:01,419 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2024-10-12 02:54:01,421 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-12 02:54:02,646 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-10-11 2024-10-12 02:54:02,781 INFO [make_ssh_files] running in process 58713 2024-10-12 02:54:02,783 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:02,783 INFO [make_ssh_files] publishing log messages to tcp://*:5563 2024-10-12 02:54:02,784 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:02,785 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-10-12 NOAA Neah Bay 00Z observation and forecast values 2024-10-12 02:54:03,299 CRITICAL [get_onc_ctd] 2024-10-11 ONC SCVIP CTD T&S file creation failed 2024-10-12 02:54:06,588 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m10d13.nc 2024-10-12 02:54:07,517 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d07.nc 2024-10-12 02:54:07,683 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d08.nc 2024-10-12 02:54:07,852 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d09.nc 2024-10-12 02:54:08,007 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d10.nc 2024-10-12 02:54:08,183 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d11.nc 2024-10-12 02:54:08,384 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d12.nc 2024-10-12 02:54:08,566 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d13.nc 2024-10-12 02:54:08,744 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d14.nc 2024-10-12 02:54:08,848 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m10d14.nc 2024-10-12 02:54:08,854 INFO [grib_to_netcdf] 2024-10-12 NEMO-atmos forcing file for forecast2 created 2024-10-12 02:54:08,857 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-10-12 02:54:08,921 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m10d15.nc 2024-10-12 02:54:09,035 INFO [make_ssh_files] sea surface height boundary file for 2024-10-12 forecast2 run created 2024-10-12 02:54:09,064 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-10-12 02:54:09,255 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-10-12 02:54:09,259 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-10-12 02:54:10,546 INFO [make_runoff_file] running in process 58773 2024-10-12 02:54:10,547 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:10,548 INFO [make_runoff_file] publishing log messages to tcp://*:5563 2024-10-12 02:54:10,549 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:10,577 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-10-12 02:54:10,667 INFO [make_runoff_file] 2024-10-12 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-10-12 02:54:10,670 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-10-12 02:54:11,078 INFO [make_v202111_runoff_file] running in process 58771 2024-10-12 02:54:11,079 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:11,080 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5565 2024-10-12 02:54:11,081 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:11,082 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-10-11 2024-10-12 02:54:12,044 ERROR [make_v202111_runoff_file] no 2024-10-11 discharge obs for Skagit_MountVernon: patching 2024-10-12 02:54:12,564 ERROR [make_v202111_runoff_file] no 2024-10-11 discharge obs for Snohomish_Monroe: patching 2024-10-12 02:54:12,762 ERROR [make_v202111_runoff_file] no 2024-10-11 discharge obs for Nisqually_McKenna: patching 2024-10-12 02:54:12,848 ERROR [make_v202111_runoff_file] no 2024-10-11 discharge obs for Greenwater_Greenwater: patching 2024-10-12 02:54:14,145 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-10-11: /results/forcing/rivers/R202108Dailies_y2024m10d11.nc 2024-10-12 02:54:14,146 INFO [make_v202111_runoff_file] 2024-10-11 runoff file creation completed 2024-10-12 02:54:14,148 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-10-12 02:54:14,351 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2024-10-12 02:54:14,357 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2024-10-12 02:54:14,362 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2024-10-12 02:54:14,366 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2024-10-12 02:54:15,599 INFO [upload_forcing] running in process 58812 2024-10-12 02:54:15,600 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:15,601 INFO [upload_forcing] publishing log messages to tcp://*:5563 2024-10-12 02:54:15,602 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:15,603 INFO [upload_forcing] running in process 58811 2024-10-12 02:54:15,604 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:15,605 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-10-12 02:54:15,606 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:15,648 INFO [upload_forcing] running in process 58814 2024-10-12 02:54:15,649 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:15,650 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-10-12 02:54:15,651 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:15,658 INFO [upload_forcing] running in process 58813 2024-10-12 02:54:15,659 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:15,660 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-10-12 02:54:15,661 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:16,719 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-10-11 2024-10-12 02:54:17,011 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m10d11.nc 2024-10-12 02:54:17,031 INFO [upload_forcing] forecast2 2024-10-12 forcing files upload to optimum-hindcast completed 2024-10-12 02:54:17,035 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-12 02:54:18,185 INFO [upload_forcing] forecast2 2024-10-12 forcing files upload to arbutus.cloud-nowcast completed 2024-10-12 02:54:18,190 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-12 02:54:18,388 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-12'], host='localhost') 2024-10-12 02:54:19,528 INFO [make_forcing_links] running in process 58841 2024-10-12 02:54:19,529 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:19,530 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-10-12 02:54:19,531 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:19,898 WARNING [get_onc_ctd] no SEVIP salinity data 2024-10-12 02:54:20,108 INFO [get_onc_ctd] 2024-10-11 ONC SEVIP CTD T&S file created 2024-10-12 02:54:20,110 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2024-10-12 02:54:20,272 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2024-10-12 02:54:20,678 INFO [make_forcing_links] forecast2 2024-10-12 forcing file links on arbutus.cloud-nowcast created 2024-10-12 02:54:20,681 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-10-12 02:54:20,888 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-12'], host='arbutus.cloud-nowcast') 2024-10-12 02:54:21,429 INFO [ping_erddap] running in process 58848 2024-10-12 02:54:21,431 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:21,432 INFO [ping_erddap] publishing log messages to tcp://*:5565 2024-10-12 02:54:21,433 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:21,434 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2024-10-12 02:54:21,436 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-12 02:54:22,766 INFO [upload_forcing] forecast2 2024-10-12 forcing files upload to robot.graham completed 2024-10-12 02:54:22,771 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-10-12 02:54:23,724 INFO [run_NEMO] running in process 22624 2024-10-12 02:54:23,726 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:23,727 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-10-12 02:54:23,786 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:24,950 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/11oct24forecast2_2024-10-12T025423.945343-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-10-12 02:54:24,968 INFO [run_NEMO] forecast2 NEMO run for 2024-10-12 on arbutus.cloud-nowcast started 2024-10-12 02:54:24,973 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-10-12 02:54:25,166 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-10-12 02:54:27,197 INFO [watch_NEMO] running in process 22876 2024-10-12 02:54:27,198 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:54:27,199 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-10-12 02:54:27,200 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:54:27,272 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-10-12 02:54:30,955 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-10-12 02:54:31,292 ERROR [upload_forcing] Traceback (most recent call last): 2024-10-12 02:54:31,294 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2327, in _check_banner 2024-10-12 02:54:31,295 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-10-12 02:54:31,296 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-12 02:54:31,298 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 381, in readline 2024-10-12 02:54:31,299 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-10-12 02:54:31,300 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-10-12 02:54:31,302 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 626, in _read_timeout 2024-10-12 02:54:31,304 ERROR [upload_forcing] raise socket.timeout() 2024-10-12 02:54:31,305 ERROR [upload_forcing] TimeoutError 2024-10-12 02:54:31,307 ERROR [upload_forcing] 2024-10-12 02:54:31,308 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-10-12 02:54:31,310 ERROR [upload_forcing] 2024-10-12 02:54:31,312 ERROR [upload_forcing] Traceback (most recent call last): 2024-10-12 02:54:31,313 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2143, in run 2024-10-12 02:54:31,315 ERROR [upload_forcing] self._check_banner() 2024-10-12 02:54:31,317 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2331, in _check_banner 2024-10-12 02:54:31,319 ERROR [upload_forcing] raise SSHException( 2024-10-12 02:54:31,321 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-10-12 02:54:31,322 ERROR [upload_forcing] 2024-10-12 02:54:31,335 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 730, in _auth key = self._key_from_filepath( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 638, in _key_from_filepath key = klass.from_private_key_file(key_path, password) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/pkey.py", line 421, in from_private_key_file key = cls(filename=filename, password=password) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/ed25519key.py", line 60, in __init__ pkformat, data = self._read_private_key("OPENSSH", f) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/pkey.py", line 529, in _read_private_key raise SSHException( paramiko.ssh_exception.SSHException: encountered RSA key, expected OPENSSH key During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 85, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 451, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 738, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2143, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2331, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-10-12 02:54:41,379 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-10-11 2024-10-12 02:55:05,754 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-10-11 2024-10-12 02:55:29,395 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-10-11 2024-10-12 02:55:46,126 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-10-11 2024-10-12 02:56:03,556 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-10-11 2024-10-12 02:56:12,507 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-10-11 2024-10-12 02:56:21,126 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-10-11; substituting empty dataset 2024-10-12 02:56:21,127 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-10-11 2024-10-12 02:56:29,364 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-10-11; substituting empty dataset 2024-10-12 02:56:29,761 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-10-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-10-12 02:56:29,778 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-10-11 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-10-12 02:58:22,320 INFO [get_onc_ferry] 2024-10-11 ONC TWDP ferry data file created 2024-10-12 02:58:22,323 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker 2024-10-12 02:58:22,524 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2024-10-12 02:58:23,677 INFO [ping_erddap] running in process 58869 2024-10-12 02:58:23,679 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 02:58:23,680 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-12 02:58:23,681 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-12 02:58:23,682 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2024-10-12 02:58:23,684 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-12 02:59:27,274 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17980290 = 2024-10-13 04:59:20 UTC, 16.6% complete 2024-10-12 03:04:27,274 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17980791 = 2024-10-13 10:33:20 UTC, 35.2% complete 2024-10-12 03:09:27,275 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17981281 = 2024-10-13 16:00:00 UTC, 53.4% complete 2024-10-12 03:14:27,277 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17981767 = 2024-10-13 21:24:00 UTC, 71.4% complete 2024-10-12 03:19:27,277 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17982224 = 2024-10-14 02:28:40 UTC, 88.3% complete 2024-10-12 03:24:27,277 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17982540 = 2024-10-14 05:59:20 UTC, 100.0% complete 2024-10-12 03:29:27,281 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2024-10-12 03:29:27,285 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-10-12 03:29:27,637 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-12'], host='arbutus.cloud-nowcast') 2024-10-12 03:29:27,646 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-12'], host='arbutus.cloud-nowcast') 2024-10-12 03:29:27,650 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-11'], host='localhost') 2024-10-12 03:29:29,113 INFO [download_results] running in process 58938 2024-10-12 03:29:29,114 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 03:29:29,115 INFO [download_results] publishing log messages to tcp://*:5563 2024-10-12 03:29:29,116 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-10-12 03:29:29,117 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/11oct24 to /results/SalishSea/forecast2.202111 2024-10-12 03:29:41,676 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-10-12 forecast2 run 2024-10-12 03:29:41,680 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-10-12 03:31:36,243 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-10-12 forecast2 run 2024-10-12 03:31:36,246 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-10-12 03:31:36,447 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-12'], host='arbutus.cloud-nowcast') 2024-10-12 03:31:47,556 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-10-12 03:31:47,762 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-10-12 03:31:59,479 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-10-12 03:36:59,529 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 90, 16.7% complete 2024-10-12 03:41:59,525 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 189, 35.0% complete 2024-10-12 03:46:59,536 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 288, 53.3% complete 2024-10-12 03:51:59,526 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 381, 70.6% complete 2024-10-12 03:54:11,544 INFO [download_results] forecast2 2024-10-11 results files from arbutus.cloud-nowcast downloaded 2024-10-12 03:54:11,548 INFO [manager] checklist updated with [results files] items from download_results worker 2024-10-12 03:54:11,710 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-10-11'], host='localhost') 2024-10-12 03:54:13,751 INFO [make_CHS_currents_file] running in process 59025 2024-10-12 03:54:13,752 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 03:54:13,754 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-10-12 03:54:13,755 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-10-12 03:54:48,655 INFO [make_CHS_currents_file] Made CHS currents file for 2024-10-11 for forecast2 2024-10-12 03:54:48,658 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-10-12 03:54:48,859 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-10-11'], host='localhost') 2024-10-12 03:54:49,989 INFO [update_forecast_datasets] running in process 59049 2024-10-12 03:54:49,990 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 03:54:49,991 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-10-12 03:54:49,992 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-10-12 03:54:49,993 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/11oct24 run 2024-10-12 03:56:58,850 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 480, 88.9% complete 2024-10-12 03:58:56,329 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/11oct24 run: /results/SalishSea/rolling-forecasts/nemo 2024-10-12 03:58:56,330 INFO [update_forecast_datasets] nemo 2024-10-11 forecast2 rolling forecast datasets updated 2024-10-12 03:58:56,332 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-10-12 03:58:56,531 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-10-12 03:58:56,537 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-10-11'], host='localhost') 2024-10-12 03:58:56,542 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-10-11'], host='localhost') 2024-10-12 03:58:57,924 INFO [ping_erddap] running in process 59125 2024-10-12 03:58:57,925 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 03:58:57,926 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-10-12 03:58:57,927 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-12 03:58:57,934 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-10-12 03:58:57,938 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-12 03:58:58,970 INFO [make_surface_current_tiles] running in process 59128 2024-10-12 03:58:58,972 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 03:58:58,972 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2024-10-12 03:58:58,973 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-10-12 03:58:58,976 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-10-11 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/11oct24 2024-10-12 03:59:01,327 INFO [make_plots] running in process 59127 2024-10-12 03:59:01,328 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 03:59:01,329 INFO [make_plots] publishing log messages to tcp://*:5565 2024-10-12 03:59:01,330 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-12 03:59:01,941 INFO [make_plots] preparing render list for 2024-10-11 NEMO forecast2 publish figures 2024-10-12 03:59:02,059 INFO [make_plots] added Website_thumbnail figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,060 INFO [make_plots] added Threshold_website figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,061 INFO [make_plots] added PA_tidal_predictions figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,062 INFO [make_plots] added Boundary Bay figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,063 INFO [make_plots] added Campbell River figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,064 INFO [make_plots] added Cherry Point figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,064 INFO [make_plots] added Friday Harbor figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,065 INFO [make_plots] added Halfmoon Bay figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,066 INFO [make_plots] added Nanaimo figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,067 INFO [make_plots] added Neah Bay figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,067 INFO [make_plots] added New Westminster figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,068 INFO [make_plots] added Patricia Bay figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,069 INFO [make_plots] added Point Atkinson figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,070 INFO [make_plots] added Port Renfrew figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,070 INFO [make_plots] added Sand Heads figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,071 INFO [make_plots] added Sandy Cove figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,072 INFO [make_plots] added Squamish figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,073 INFO [make_plots] added Victoria figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,073 INFO [make_plots] added Woodwards Landing figure to 2024-10-11 NEMO forecast2 publish render list 2024-10-12 03:59:02,075 INFO [make_plots] starting to render nemo forecast2 publish 11oct24 figures 2024-10-12 04:01:58,850 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2024-10-12 04:01:58,854 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-10-12 04:01:59,251 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-10-12'], host='localhost') 2024-10-12 04:02:00,816 INFO [download_wwatch3_results] running in process 59701 2024-10-12 04:02:00,818 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 04:02:00,819 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566 2024-10-12 04:02:00,821 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-10-12 04:04:14,650 INFO [download_wwatch3_results] forecast2 2024-10-12 results files from arbutus.cloud-nowcast downloaded 2024-10-12 04:04:14,653 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-10-12 04:04:15,006 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-10-12'], host='localhost') 2024-10-12 04:04:16,583 INFO [update_forecast_datasets] running in process 60136 2024-10-12 04:04:16,584 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 04:04:16,585 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566 2024-10-12 04:04:16,586 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-10-12 04:04:16,586 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/12oct24 run 2024-10-12 04:04:16,588 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/12oct24 run 2024-10-12 04:04:46,965 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/12oct24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-10-12 04:04:46,967 INFO [update_forecast_datasets] wwatch3 2024-10-12 forecast2 rolling forecast datasets updated 2024-10-12 04:04:46,969 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-10-12 04:04:47,362 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-10-12 04:04:49,038 INFO [ping_erddap] running in process 60214 2024-10-12 04:04:49,039 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 04:04:49,040 INFO [ping_erddap] publishing log messages to tcp://*:5566 2024-10-12 04:04:49,041 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-10-12 04:04:49,042 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-10-12 04:04:49,043 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-10-12 04:04:49,423 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-10-12'], host='localhost') 2024-10-12 04:04:54,335 INFO [make_plots] running in process 60220 2024-10-12 04:04:54,336 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 04:04:54,337 INFO [make_plots] publishing log messages to tcp://*:5566 2024-10-12 04:04:54,338 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-10-12 04:04:54,339 INFO [make_plots] preparing render list for 2024-10-12 WaveWatch3 forecast2 publish figures 2024-10-12 04:04:54,339 INFO [make_plots] added Halibut Bank figure to 2024-10-12 WaveWatch3 forecast2 publish render list 2024-10-12 04:04:54,340 INFO [make_plots] added Sentry Shoal figure to 2024-10-12 WaveWatch3 forecast2 publish render list 2024-10-12 04:04:54,341 INFO [make_plots] starting to render wwatch3 forecast2 publish 12oct24 figures 2024-10-12 04:04:54,807 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-10-12 04:05:04,315 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-10-12 04:05:11,426 INFO [make_plots] finished rendering wwatch3 forecast2 publish 12oct24 figures 2024-10-12 04:05:11,427 INFO [make_plots] wwatch3 publish plots for 2024-10-12 forecast2 completed 2024-10-12 04:05:11,430 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-12 04:10:21,105 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/11oct24 2024-10-12 04:12:39,930 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-10-12 04:12:39,931 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-10-12 04:13:15,148 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:14:39,678 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/11oct24 2024-10-12 04:14:39,688 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-10-11 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/11oct24 2024-10-12 04:14:39,689 INFO [make_surface_current_tiles] surface current tile figures for 2024-10-11 forecast2 completed 2024-10-12 04:14:39,958 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-10-12 04:15:01,967 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:15:02,167 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-10-12 04:15:02,850 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:16:11,742 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:16:46,865 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:17:25,695 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:18:01,143 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:18:35,666 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:19:11,151 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:19:46,361 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:19:46,537 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-10-12 04:19:46,764 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:20:21,747 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-10-07 00:00:00Z to 2024-10-14 05:55:00Z 2024-10-12 04:21:01,325 INFO [make_plots] finished rendering nemo forecast2 publish 11oct24 figures 2024-10-12 04:21:01,326 INFO [make_plots] nemo publish plots for 2024-10-11 forecast2 completed 2024-10-12 04:21:01,334 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-10-12 04:21:01,663 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-10-11'], host='localhost') 2024-10-12 04:21:04,373 INFO [make_feeds] running in process 61176 2024-10-12 04:21:04,374 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 04:21:04,375 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-10-12 04:21:04,377 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-10-12 04:21:31,394 INFO [make_feeds] ATOM feeds for 2024-10-11 forecast2 run completed 2024-10-12 04:21:31,397 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-10-12 04:21:31,668 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2024-10-12 04:21:32,768 INFO [clear_checklist] running in process 61200 2024-10-12 04:21:32,769 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 04:21:32,770 INFO [clear_checklist] publishing log messages to tcp://*:5563 2024-10-12 04:21:32,771 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2024-10-12 04:21:32,772 INFO [clear_checklist] requesting that manager clear system state checklist 2024-10-12 04:21:32,774 INFO [manager] writing checklist to log file 2024-10-12 04:21:32,819 INFO [manager] checklist cleared 2024-10-12 04:21:32,820 INFO [clear_checklist] nowcast system checklist cleared 2024-10-12 04:21:32,822 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2024-10-12 04:21:33,845 INFO [rotate_logs] running in process 61205 2024-10-12 04:21:33,846 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-10-12 04:21:33,847 INFO [rotate_logs] publishing log messages to tcp://*:5563 2024-10-12 04:21:33,849 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2024-10-12 04:21:33,850 INFO [rotate_logs] rotating log files