2024-11-01 04:18:28,434 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2024-11-01 04:18:28,442 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2024-11-01 04:18:28,447 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2024-11-01 04:18:28,449 INFO [rotate_logs] log files rotated 2024-11-01 04:18:28,455 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2024-11-01 04:37:00,400 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 04:37:00,554 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-02-21'], host='localhost') 2024-11-01 04:37:00,561 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 04:37:00,567 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 04:37:01,844 INFO [download_results] running in process 384394 2024-11-01 04:37:01,845 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 04:37:01,846 INFO [download_results] publishing log messages to tcp://*:5565 2024-11-01 04:37:01,847 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 04:37:01,848 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21feb12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 04:37:06,150 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 04:44:52,988 INFO [download_results] hindcast 2012-02-21 results files from optimum-hindcast downloaded 2024-11-01 04:44:53,007 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 04:44:53,017 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-02-21'], host='localhost') 2024-11-01 04:44:54,867 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 05:47:09,370 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 05:47:09,530 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-02-26'], host='localhost') 2024-11-01 05:47:09,537 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 05:47:09,542 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 05:47:10,811 INFO [download_results] running in process 384868 2024-11-01 05:47:10,812 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 05:47:10,813 INFO [download_results] publishing log messages to tcp://*:5565 2024-11-01 05:47:10,814 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 05:47:10,816 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26feb12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 05:47:15,016 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 05:53:29,887 INFO [download_results] hindcast 2012-02-26 results files from optimum-hindcast downloaded 2024-11-01 05:53:29,905 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 05:53:29,913 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-02-26'], host='localhost') 2024-11-01 05:53:31,360 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 05:53:31,371 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2012-feb', 'robot.graham'], host='localhost') 2024-11-01 05:53:32,643 INFO [archive_tarball] running in process 384922 2024-11-01 05:53:32,644 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 05:53:32,645 INFO [archive_tarball] publishing log messages to tcp://*:5564 2024-11-01 05:53:32,646 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-11-01 05:53:32,647 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb12.tar from /ocean/sallen/timbucktwo/oxygen/*feb12/ 2024-11-01 07:07:32,971 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb12.index from /ocean/dlatorne/oxygen-feb12.tar 2024-11-01 07:08:07,915 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-feb12.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-11-01 07:17:19,812 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 07:17:20,000 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-03-01'], host='localhost') 2024-11-01 07:17:20,007 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 07:17:20,013 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 07:17:22,132 INFO [download_results] running in process 387031 2024-11-01 07:17:22,133 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 07:17:22,134 INFO [download_results] publishing log messages to tcp://*:5565 2024-11-01 07:17:22,135 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 07:17:22,136 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01mar12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 07:17:26,241 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 07:25:45,483 INFO [download_results] hindcast 2012-03-01 results files from optimum-hindcast downloaded 2024-11-01 07:25:45,499 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 07:25:45,510 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-03-01'], host='localhost') 2024-11-01 07:25:47,659 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 08:42:31,496 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 08:42:31,665 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-03-06'], host='localhost') 2024-11-01 08:42:31,672 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 08:42:31,677 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 08:42:33,612 INFO [download_results] running in process 388156 2024-11-01 08:42:33,613 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:42:33,614 INFO [download_results] publishing log messages to tcp://*:5562 2024-11-01 08:42:33,615 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:42:33,616 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06mar12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 08:42:37,773 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 08:50:53,733 INFO [download_results] hindcast 2012-03-06 results files from optimum-hindcast downloaded 2024-11-01 08:50:53,751 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 08:50:53,762 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-03-06'], host='localhost') 2024-11-01 08:50:55,704 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 08:55:24,367 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20241101/12/ 2024-11-01 08:55:24,368 INFO [collect_weather] 2024-11-01 2.5km weather forecast 12 collection complete 2024-11-01 08:55:24,371 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-11-01 08:55:24,524 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-10-31'], host='localhost') 2024-11-01 08:55:24,530 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-10-31'], host='localhost') 2024-11-01 08:55:24,535 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-10-31'], host='localhost') 2024-11-01 08:55:24,541 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-10-31'], host='localhost') 2024-11-01 08:55:24,547 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2024-11-01 08:55:24,552 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2024-11-01 08:55:24,556 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2024-11-01 08:55:24,562 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2024-11-01 08:55:24,566 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2024-11-01 08:55:26,485 INFO [collect_weather] running in process 388239 2024-11-01 08:55:26,486 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:26,487 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-11-01 08:55:26,488 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:26,490 INFO [collect_NeahBay_ssh] running in process 388237 2024-11-01 08:55:26,491 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:26,492 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563 2024-11-01 08:55:26,493 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:26,494 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-11-01 2024-11-01 08:55:26,508 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2024-11-01 08:55:26,626 INFO [download_live_ocean] running in process 388238 2024-11-01 08:55:26,628 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:26,629 INFO [download_live_ocean] publishing log messages to tcp://*:5565 2024-11-01 08:55:26,629 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:26,630 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-11-01 2024-11-01 08:55:27,124 INFO [collect_river_data] running in process 388231 2024-11-01 08:55:27,125 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:27,126 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-11-01 08:55:27,127 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:27,128 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-10-31 2024-11-01 08:55:27,148 INFO [collect_river_data] running in process 388233 2024-11-01 08:55:27,149 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:27,150 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-11-01 08:55:27,151 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:27,152 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-10-31 2024-11-01 08:55:27,157 INFO [download_live_ocean] downloaded 10191317 bytes from https://liveocean.apl.uw.edu/output/f2024.11.01/ubc.nc to /results/forcing/LiveOcean/downloaded/20241101/low_passed_UBC.nc 2024-11-01 08:55:27,158 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2024-11-01 08:55:27,169 INFO [collect_river_data] running in process 388234 2024-11-01 08:55:27,170 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:27,171 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-11-01 08:55:27,172 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:27,172 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-10-31 2024-11-01 08:55:27,192 INFO [collect_river_data] running in process 388235 2024-11-01 08:55:27,193 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:27,194 INFO [collect_river_data] publishing log messages to tcp://*:5571 2024-11-01 08:55:27,195 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:27,196 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-10-31 2024-11-01 08:55:27,374 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2024-10-31&endDT=2024-10-31 "HTTP/1.1 200 OK" 2024-11-01 08:55:27,403 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-10-31 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2024-11-01 08:55:27,404 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-10-31 completed 2024-11-01 08:55:27,407 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-01 08:55:27,478 INFO [make_turbidity_file] running in process 388236 2024-11-01 08:55:27,479 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:27,481 INFO [make_turbidity_file] publishing log messages to tcp://*:5572 2024-11-01 08:55:27,482 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:27,483 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-11-01 2024-11-01 08:55:27,485 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2024-10-31&endDT=2024-10-31 "HTTP/1.1 200 OK" 2024-11-01 08:55:27,489 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2024-10-31&endDT=2024-10-31 "HTTP/1.1 200 OK" 2024-11-01 08:55:27,493 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-10-31 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2024-11-01 08:55:27,494 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-10-31 completed 2024-11-01 08:55:27,497 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2024-10-31&endDT=2024-10-31 "HTTP/1.1 200 OK" 2024-11-01 08:55:27,497 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-01 08:55:27,504 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-10-31 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2024-11-01 08:55:27,505 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-10-31 completed 2024-11-01 08:55:27,508 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-01 08:55:27,544 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-10-31 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2024-11-01 08:55:27,545 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-10-31 completed 2024-11-01 08:55:27,546 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-01 08:55:27,860 INFO [crop_gribs] running in process 388240 2024-11-01 08:55:27,861 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:27,862 INFO [crop_gribs] publishing log messages to tcp://*:5568 2024-11-01 08:55:27,863 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:27,865 INFO [crop_gribs] cropping 2024-11-01 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2024-11-01 08:55:27,877 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241101/18/ 2024-11-01 08:55:28,177 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20241101/low_passed_UBC.nc 2024-11-01 08:55:28,178 INFO [download_live_ocean] 2024-11-01 Live Ocean file for Salish Sea western boundary downloaded 2024-11-01 08:55:28,180 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2024-11-01 08:55:28,336 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-11-01'], host='localhost') 2024-11-01 08:55:30,548 INFO [make_turbidity_file] 2024-11-01 Fraser River turbidity file creation complete 2024-11-01 08:55:30,550 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2024-11-01 08:55:31,299 INFO [make_live_ocean_files] running in process 388391 2024-11-01 08:55:31,301 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:31,302 INFO [make_live_ocean_files] publishing log messages to tcp://*:5565 2024-11-01 08:55:31,303 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:31,304 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-11-01 Live Ocean run 2024-11-01 08:55:32,537 INFO [collect_NeahBay_ssh] 2024-11-01T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2024-11-01 08:55:32,539 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-11-01 08:55:32,547 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 08:55:35,339 INFO [make_ssh_files] running in process 388415 2024-11-01 08:55:35,340 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:35,342 INFO [make_ssh_files] publishing log messages to tcp://*:5563 2024-11-01 08:55:35,343 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:35,344 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-11-01 NOAA Neah Bay 06Z observation and forecast values 2024-11-01 08:55:39,963 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d28.nc 2024-11-01 08:55:40,147 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d29.nc 2024-11-01 08:55:40,326 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d30.nc 2024-11-01 08:55:40,502 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d31.nc 2024-11-01 08:55:40,720 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d01.nc 2024-11-01 08:55:40,895 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d02.nc 2024-11-01 08:55:41,080 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d03.nc 2024-11-01 08:55:41,259 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d04.nc 2024-11-01 08:55:41,391 INFO [make_ssh_files] sea surface height boundary file for 2024-11-01 nowcast run created 2024-11-01 08:55:41,396 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-11-01 08:55:41,408 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-11-01 08:55:41,415 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-11-01 08:55:42,817 INFO [make_runoff_file] running in process 388448 2024-11-01 08:55:42,818 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:42,819 INFO [make_runoff_file] publishing log messages to tcp://*:5563 2024-11-01 08:55:42,820 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:42,842 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-11-01 08:55:42,966 INFO [make_runoff_file] 2024-11-01 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-11-01 08:55:42,969 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-11-01 08:55:43,400 INFO [make_v202111_runoff_file] running in process 388446 2024-11-01 08:55:43,401 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 08:55:43,402 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5563 2024-11-01 08:55:43,403 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-11-01 08:55:43,404 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-10-31 2024-11-01 08:55:46,504 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-10-31: /results/forcing/rivers/R202108Dailies_y2024m10d31.nc 2024-11-01 08:55:46,505 INFO [make_v202111_runoff_file] 2024-10-31 runoff file creation completed 2024-11-01 08:55:46,507 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-11-01 08:56:13,888 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m11d01.nc 2024-11-01 08:56:13,889 INFO [make_live_ocean_files] 2024-11-01 Live Ocean western boundary conditions files created 2024-11-01 08:56:13,891 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2024-11-01 10:07:42,765 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 10:07:42,981 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-03-11'], host='localhost') 2024-11-01 10:07:42,990 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 10:07:42,995 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 10:07:45,123 INFO [download_results] running in process 389330 2024-11-01 10:07:45,124 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:07:45,125 INFO [download_results] publishing log messages to tcp://*:5567 2024-11-01 10:07:45,127 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:07:45,128 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11mar12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 10:07:49,240 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 10:16:02,354 INFO [download_results] hindcast 2012-03-11 results files from optimum-hindcast downloaded 2024-11-01 10:16:02,369 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 10:16:02,381 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-03-11'], host='localhost') 2024-11-01 10:16:04,508 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 10:55:41,778 INFO [crop_gribs] crop_gribs 12 has watched for 8h; retrying remaining 1 file(s) 2024-11-01 10:55:44,515 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241101/12/ 2024-11-01 10:55:44,516 INFO [crop_gribs] 2024-11-01 12 GRIBs cropping complete 2024-11-01 10:55:44,518 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-11-01 10:55:44,675 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2024-11-01 10:55:47,602 INFO [grib_to_netcdf] running in process 389875 2024-11-01 10:55:47,603 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:55:47,604 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565 2024-11-01 10:55:47,605 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:55:47,606 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-11-01 nowcast and forecast runs 2024-11-01 10:55:59,910 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m11d01.nc 2024-11-01 10:56:08,785 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m11d02.nc 2024-11-01 10:56:14,053 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m11d03.nc 2024-11-01 10:56:14,068 INFO [grib_to_netcdf] 2024-11-01 NEMO-atmos forcing file for nowcast+ created 2024-11-01 10:56:14,072 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-11-01 10:56:14,089 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2024-11-01 10:56:14,094 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2024-11-01 10:56:14,098 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2024-11-01 10:56:14,103 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2024-11-01 10:56:14,109 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2024-11-01 10:56:15,440 INFO [upload_forcing] running in process 389914 2024-11-01 10:56:15,443 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:56:15,444 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-11-01 10:56:15,445 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:56:15,449 INFO [ping_erddap] running in process 389916 2024-11-01 10:56:15,450 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:56:15,452 INFO [ping_erddap] publishing log messages to tcp://*:5566 2024-11-01 10:56:15,453 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:56:15,455 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2024-11-01 10:56:15,456 INFO [upload_forcing] running in process 389911 2024-11-01 10:56:15,457 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:56:15,458 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-01 10:56:15,459 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-11-01 10:56:15,459 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:56:15,460 INFO [upload_forcing] running in process 389913 2024-11-01 10:56:15,461 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:56:15,462 INFO [upload_forcing] publishing log messages to tcp://*:5569 2024-11-01 10:56:15,463 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:56:15,480 INFO [upload_forcing] running in process 389915 2024-11-01 10:56:15,481 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:56:15,482 INFO [upload_forcing] publishing log messages to tcp://*:5570 2024-11-01 10:56:15,483 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:56:18,142 INFO [upload_forcing] nowcast+ 2024-11-01 forcing files upload to optimum-hindcast completed 2024-11-01 10:56:18,145 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-01 10:56:18,148 INFO [upload_forcing] nowcast+ 2024-11-01 forcing files upload to orcinus-nowcast-agrif completed 2024-11-01 10:56:18,161 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-01 10:56:25,847 INFO [upload_forcing] nowcast+ 2024-11-01 forcing files upload to robot.graham completed 2024-11-01 10:56:25,852 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-01 10:56:30,448 INFO [upload_forcing] nowcast+ 2024-11-01 forcing files upload to arbutus.cloud-nowcast completed 2024-11-01 10:56:30,453 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-01 10:56:30,471 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 10:56:31,831 INFO [make_forcing_links] running in process 389953 2024-11-01 10:56:31,832 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:56:31,833 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-11-01 10:56:31,834 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:56:33,010 INFO [make_forcing_links] nowcast+ 2024-11-01 forcing file links on arbutus.cloud-nowcast created 2024-11-01 10:56:33,013 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-11-01 10:56:33,027 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-11-01'], host='arbutus.cloud-nowcast') 2024-11-01 10:56:35,305 INFO [run_NEMO] running in process 13677 2024-11-01 10:56:35,306 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:56:35,307 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-11-01 10:56:35,366 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:56:36,215 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/01nov24nowcast_2024-11-01T105635.354228-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-11-01 10:56:36,234 INFO [run_NEMO] nowcast NEMO run for 2024-11-01 on arbutus.cloud-nowcast started 2024-11-01 10:56:36,239 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-11-01 10:56:36,255 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-11-01 10:56:38,242 INFO [watch_NEMO] running in process 13929 2024-11-01 10:56:38,244 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 10:56:38,245 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-11-01 10:56:38,246 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-11-01 10:56:38,312 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-11-01 11:01:38,313 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18021318 = 2024-11-01 04:51:20 UTC, 20.2% complete 2024-11-01 11:06:38,315 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18021843 = 2024-11-01 10:41:20 UTC, 44.6% complete 2024-11-01 11:11:38,316 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18022346 = 2024-11-01 16:16:40 UTC, 67.9% complete 2024-11-01 11:16:38,317 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18022811 = 2024-11-01 21:26:40 UTC, 89.4% complete 2024-11-01 11:21:38,322 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2024-11-01 11:21:38,325 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-11-01 11:21:38,482 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 11:21:38,489 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 11:21:40,237 INFO [make_forcing_links] running in process 390331 2024-11-01 11:21:40,239 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 11:21:40,240 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-11-01 11:21:40,241 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-11-01 11:21:40,242 INFO [download_results] running in process 390332 2024-11-01 11:21:40,243 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 11:21:40,244 INFO [download_results] publishing log messages to tcp://*:5566 2024-11-01 11:21:40,245 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 11:21:40,246 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/01nov24 to /results/SalishSea/nowcast-blue.202111 2024-11-01 11:21:41,387 INFO [make_forcing_links] ssh 2024-11-01 forcing file links on arbutus.cloud-nowcast created 2024-11-01 11:21:41,392 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-11-01 11:21:41,409 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-11-01'], host='arbutus.cloud-nowcast') 2024-11-01 11:21:44,921 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/01nov24forecast_2024-11-01T112143.970529-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-11-01 11:21:44,939 INFO [run_NEMO] forecast NEMO run for 2024-11-01 on arbutus.cloud-nowcast started 2024-11-01 11:21:44,943 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-11-01 11:21:44,960 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-11-01 11:21:48,585 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-11-01 11:26:48,525 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18023496 = 2024-11-02 05:03:20 UTC, 14.0% complete 2024-11-01 11:31:48,521 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18024012 = 2024-11-02 10:47:20 UTC, 30.0% complete 2024-11-01 11:36:48,547 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18024505 = 2024-11-02 16:16:00 UTC, 45.2% complete 2024-11-01 11:37:54,303 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 11:37:54,464 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-03-16'], host='localhost') 2024-11-01 11:37:54,470 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 11:37:54,476 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 11:37:55,847 INFO [download_results] running in process 390447 2024-11-01 11:37:55,848 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 11:37:55,849 INFO [download_results] publishing log messages to tcp://*:5565 2024-11-01 11:37:55,850 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 11:37:55,851 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16mar12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 11:37:59,990 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 11:41:48,532 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18024964 = 2024-11-02 21:22:00 UTC, 59.4% complete 2024-11-01 11:41:58,274 INFO [download_results] nowcast 2024-11-01 results files from arbutus.cloud-nowcast downloaded 2024-11-01 11:41:58,280 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 11:41:58,312 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 11:41:58,334 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-10-31'], host='localhost') 2024-11-01 11:41:58,349 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 11:42:05,079 INFO [make_CHS_currents_file] running in process 390529 2024-11-01 11:42:05,080 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 11:42:05,082 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-11-01 11:42:05,084 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-11-01 11:42:08,655 INFO [make_plots] running in process 390526 2024-11-01 11:42:08,656 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 11:42:08,658 INFO [make_plots] publishing log messages to tcp://*:5566 2024-11-01 11:42:08,665 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-11-01 11:42:08,712 INFO [make_plots] running in process 390528 2024-11-01 11:42:08,714 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 11:42:08,716 INFO [make_plots] publishing log messages to tcp://*:5569 2024-11-01 11:42:08,722 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-11-01 11:42:09,739 INFO [make_plots] preparing render list for 2024-10-31 NEMO nowcast-blue comparison figures 2024-11-01 11:42:09,745 INFO [make_plots] preparing render list for 2024-11-01 NEMO nowcast-blue research figures 2024-11-01 11:42:11,240 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-blue research render list 2024-11-01 11:42:12,459 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-blue research render list 2024-11-01 11:42:12,466 INFO [make_plots] added Currents_sections_and_surface figure to 2024-11-01 NEMO nowcast-blue research render list 2024-11-01 11:42:12,474 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-11-01 NEMO nowcast-blue research render list 2024-11-01 11:42:12,481 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-11-01 NEMO nowcast-blue research render list 2024-11-01 11:42:12,486 INFO [make_plots] starting to render nemo nowcast research 01nov24 figures 2024-11-01 11:42:47,474 INFO [make_plots] added SH_wind figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,477 INFO [make_plots] added Compare_VENUS_East figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,481 INFO [make_plots] added Compare_VENUS_Central figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,487 INFO [make_plots] added Central_ADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,491 INFO [make_plots] added Central_depavADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,496 INFO [make_plots] added Central_timeavADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,500 INFO [make_plots] added East_ADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,503 INFO [make_plots] added East_depavADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,507 INFO [make_plots] added East_timeavADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,510 INFO [make_plots] added ddl_ADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,513 INFO [make_plots] added ddl_depavADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,517 INFO [make_plots] added ddl_timeavADCP figure to 2024-10-31 NEMO nowcast-blue comparison render list 2024-11-01 11:42:47,520 INFO [make_plots] starting to render nemo nowcast comparison 31oct24 figures 2024-11-01 11:42:48,877 INFO [make_CHS_currents_file] Made CHS currents file for 2024-11-01 for nowcast 2024-11-01 11:42:48,881 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-11-01 11:43:40,867 INFO [make_plots] finished rendering nemo nowcast research 01nov24 figures 2024-11-01 11:43:40,870 INFO [make_plots] nemo research plots for 2024-11-01 nowcast completed 2024-11-01 11:43:40,885 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-11-01 11:44:21,460 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available 2024-11-01 11:44:22,071 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-11-01 11:44:22,653 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-11-01 11:44:23,227 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-11-01 11:44:23,819 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-11-01 11:44:24,394 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-11-01 11:44:24,968 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-11-01 11:44:25,538 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-11-01 11:44:26,104 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-11-01 11:44:26,668 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-11-01 11:44:26,670 INFO [make_plots] finished rendering nemo nowcast comparison 31oct24 figures 2024-11-01 11:44:26,771 INFO [make_plots] nemo comparison plots for 2024-10-31 nowcast completed 2024-11-01 11:44:26,774 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-11-01 11:46:26,571 INFO [download_results] hindcast 2012-03-16 results files from optimum-hindcast downloaded 2024-11-01 11:46:26,587 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 11:46:26,606 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-03-16'], host='localhost') 2024-11-01 11:46:28,395 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 11:46:48,497 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18025405 = 2024-11-03 02:16:00 UTC, 73.0% complete 2024-11-01 11:51:48,498 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18025878 = 2024-11-03 07:31:20 UTC, 87.6% complete 2024-11-01 11:56:48,499 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18026280 = 2024-11-03 11:59:20 UTC, 100.0% complete 2024-11-01 12:01:48,504 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2024-11-01 12:01:48,507 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-11-01 12:01:48,682 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-11-01'], host='arbutus.cloud-nowcast') 2024-11-01 12:01:48,690 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-11-01'], host='arbutus.cloud-nowcast') 2024-11-01 12:01:48,694 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:01:48,700 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:01:48,705 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:01:48,713 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:01:48,720 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:01:50,120 INFO [download_results] running in process 390816 2024-11-01 12:01:50,121 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:01:50,122 INFO [download_results] publishing log messages to tcp://*:5563 2024-11-01 12:01:50,123 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:01:50,124 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/01nov24 to /results/SalishSea/forecast.202111 2024-11-01 12:01:50,126 INFO [upload_forcing] running in process 390812 2024-11-01 12:01:50,126 INFO [upload_forcing] running in process 390811 2024-11-01 12:01:50,127 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:01:50,128 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:01:50,129 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-11-01 12:01:50,130 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-11-01 12:01:50,131 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:01:50,132 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:01:50,151 INFO [upload_forcing] running in process 390813 2024-11-01 12:01:50,152 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:01:50,154 INFO [upload_forcing] publishing log messages to tcp://*:5569 2024-11-01 12:01:50,155 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:01:50,158 INFO [upload_forcing] running in process 390815 2024-11-01 12:01:50,159 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:01:50,160 INFO [upload_forcing] publishing log messages to tcp://*:5570 2024-11-01 12:01:50,161 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:01:50,899 INFO [upload_forcing] turbidity 2024-11-01 forcing files upload to optimum-hindcast completed 2024-11-01 12:01:50,903 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-01 12:01:51,052 INFO [upload_forcing] turbidity 2024-11-01 forcing files upload to orcinus-nowcast-agrif completed 2024-11-01 12:01:51,056 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-01 12:01:51,080 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:01:51,157 INFO [upload_forcing] turbidity 2024-11-01 forcing files upload to arbutus.cloud-nowcast completed 2024-11-01 12:01:51,162 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-01 12:01:51,185 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:01:52,597 INFO [make_forcing_links] running in process 390849 2024-11-01 12:01:52,598 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:01:52,599 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-11-01 12:01:52,600 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:01:52,670 INFO [make_forcing_links] running in process 390851 2024-11-01 12:01:52,671 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:01:52,672 INFO [make_forcing_links] publishing log messages to tcp://*:5566 2024-11-01 12:01:52,673 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:01:53,354 INFO [make_forcing_links] nowcast-agrif 2024-11-01 forcing file links on orcinus-nowcast-agrif created 2024-11-01 12:01:53,358 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-11-01 12:01:53,380 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:01:53,656 INFO [upload_forcing] turbidity 2024-11-01 forcing files upload to robot.graham completed 2024-11-01 12:01:53,661 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-01 12:01:54,533 INFO [run_NEMO_agrif] running in process 390864 2024-11-01 12:01:54,534 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:01:54,535 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5565 2024-11-01 12:01:54,536 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:02:02,354 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-11-01 forecast run 2024-11-01 12:02:02,358 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-11-01 12:02:12,027 INFO [make_forcing_links] nowcast-green 2024-11-01 forcing file links on arbutus.cloud-nowcast created 2024-11-01 12:02:12,031 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-11-01 12:02:12,056 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-11-01'], host='arbutus.cloud-nowcast') 2024-11-01 12:02:14,618 ERROR [run_NEMO_agrif] Exception (client): Error reading SSH protocol banner 2024-11-01 12:02:15,122 ERROR [run_NEMO_agrif] Traceback (most recent call last): 2024-11-01 12:02:15,124 ERROR [run_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2327, in _check_banner 2024-11-01 12:02:15,126 ERROR [run_NEMO_agrif] buf = self.packetizer.readline(timeout) 2024-11-01 12:02:15,129 ERROR [run_NEMO_agrif] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-01 12:02:15,131 ERROR [run_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 381, in readline 2024-11-01 12:02:15,133 ERROR [run_NEMO_agrif] buf += self._read_timeout(timeout) 2024-11-01 12:02:15,135 ERROR [run_NEMO_agrif] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-01 12:02:15,137 ERROR [run_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 626, in _read_timeout 2024-11-01 12:02:15,139 ERROR [run_NEMO_agrif] raise socket.timeout() 2024-11-01 12:02:15,141 ERROR [run_NEMO_agrif] TimeoutError 2024-11-01 12:02:15,142 ERROR [run_NEMO_agrif] 2024-11-01 12:02:15,144 ERROR [run_NEMO_agrif] During handling of the above exception, another exception occurred: 2024-11-01 12:02:15,145 ERROR [run_NEMO_agrif] 2024-11-01 12:02:15,147 ERROR [run_NEMO_agrif] Traceback (most recent call last): 2024-11-01 12:02:15,149 ERROR [run_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2143, in run 2024-11-01 12:02:15,151 ERROR [run_NEMO_agrif] self._check_banner() 2024-11-01 12:02:15,153 ERROR [run_NEMO_agrif] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2331, in _check_banner 2024-11-01 12:02:15,155 ERROR [run_NEMO_agrif] raise SSHException( 2024-11-01 12:02:15,157 ERROR [run_NEMO_agrif] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-11-01 12:02:15,159 ERROR [run_NEMO_agrif] 2024-11-01 12:02:17,055 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/01nov24nowcast-green_2024-11-01T120215.909716-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-11-01 12:02:17,074 INFO [run_NEMO] nowcast-green NEMO run for 2024-11-01 on arbutus.cloud-nowcast started 2024-11-01 12:02:17,079 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-11-01 12:02:17,101 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2024-11-01 12:02:20,103 INFO [run_NEMO_agrif] job id for 01nov24nowcast-agrif: 11410650.orca2.ibb 2024-11-01 12:02:20,104 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-11-01 queued on orcinus-nowcast-agrif 2024-11-01 12:02:20,110 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker 2024-11-01 12:02:20,132 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11410650.orca2.ibb'], host='localhost') 2024-11-01 12:02:21,310 INFO [watch_NEMO_agrif] running in process 390883 2024-11-01 12:02:21,311 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:02:21,312 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5565 2024-11-01 12:02:21,313 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:02:21,939 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-11-01 12:02:50,530 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-11-01 forecast run 2024-11-01 12:02:50,535 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-11-01 12:02:50,562 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-11-01'], host='arbutus.cloud-nowcast') 2024-11-01 12:02:53,075 INFO [run_ww3] running in process 16974 2024-11-01 12:02:53,077 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:02:53,078 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-11-01 12:02:53,079 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:02:53,081 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-11-01T120253.022314-0700 2024-11-01 12:02:53,130 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-11-01T120253.022314-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-11-01 12:02:53,132 INFO [run_ww3] nowcast WaveWatch3 run for 2024-11-01 on arbutus.cloud-nowcast started 2024-11-01 12:02:53,246 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-11-01 12:02:53,269 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-11-01 12:02:55,925 INFO [watch_ww3] running in process 17060 2024-11-01 12:02:55,927 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:02:55,928 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-11-01 12:02:55,929 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:02:56,133 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-11-01 12:04:32,081 CRITICAL [watch_NEMO_agrif] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 730, in _auth key = self._key_from_filepath( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 638, in _key_from_filepath key = klass.from_private_key_file(key_path, password) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/pkey.py", line 421, in from_private_key_file key = cls(filename=filename, password=password) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/ed25519key.py", line 60, in __init__ pkformat, data = self._read_private_key("OPENSSH", f) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/pkey.py", line 529, in _read_private_key raise SSHException( paramiko.ssh_exception.SSHException: encountered RSA key, expected OPENSSH key During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/watch_NEMO_agrif.py", line 91, in watch_NEMO_agrif ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 85, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 386, in connect sock.connect(addr) TimeoutError: [Errno 110] Connection timed out During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/watch_NEMO_agrif.py", line 100, in watch_NEMO_agrif sftp_client.close() ^^^^^^^^^^^ UnboundLocalError: cannot access local variable 'sftp_client' where it is not associated with a value 2024-11-01 12:07:21,832 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18021048 = 2024-11-01 01:51:20 UTC, 7.7% complete 2024-11-01 12:07:56,121 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 99, 22.9% complete 2024-11-01 12:12:21,796 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18021242 = 2024-11-01 04:00:40 UTC, 16.7% complete 2024-11-01 12:12:56,119 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 201, 46.5% complete 2024-11-01 12:13:25,626 INFO [watch_NEMO_agrif] running in process 390941 2024-11-01 12:13:25,627 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:13:25,629 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5565 2024-11-01 12:13:25,630 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:13:26,461 INFO [watch_NEMO_agrif] watching 01nov24nowcast-agrif job 11410650 on orcinus-nowcast-agrif 2024-11-01 12:13:27,417 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8042001 = 2024-11-01 03:33:20 UTC, 14.8% complete 2024-11-01 12:17:21,803 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18021434 = 2024-11-01 06:08:40 UTC, 25.6% complete 2024-11-01 12:17:56,107 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 306, 70.8% complete 2024-11-01 12:18:27,894 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8042207 = 2024-11-01 05:50:40 UTC, 24.4% complete 2024-11-01 12:22:21,801 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18021618 = 2024-11-01 08:11:20 UTC, 34.1% complete 2024-11-01 12:22:56,092 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 417, 96.5% complete 2024-11-01 12:23:28,373 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8042413 = 2024-11-01 08:08:00 UTC, 33.9% complete 2024-11-01 12:27:21,810 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18021809 = 2024-11-01 10:18:40 UTC, 43.0% complete 2024-11-01 12:27:56,136 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-11-01 12:27:56,138 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-11-01 12:27:56,318 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:27:56,369 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-11-01'], host='arbutus.cloud-nowcast') 2024-11-01 12:27:58,056 INFO [download_wwatch3_results] running in process 391321 2024-11-01 12:27:58,078 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:27:58,095 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566 2024-11-01 12:27:58,097 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:27:59,939 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-11-01 12:27:59,963 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-11-01 12:28:03,413 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-11-01 12:28:28,861 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8042621 = 2024-11-01 10:26:40 UTC, 43.5% complete 2024-11-01 12:32:13,338 INFO [download_wwatch3_results] nowcast 2024-11-01 results files from arbutus.cloud-nowcast downloaded 2024-11-01 12:32:13,342 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-11-01 12:32:21,812 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18022001 = 2024-11-01 12:26:40 UTC, 51.9% complete 2024-11-01 12:33:03,352 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 99, 15.3% complete 2024-11-01 12:33:29,345 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8042828 = 2024-11-01 12:44:40 UTC, 53.1% complete 2024-11-01 12:33:41,091 INFO [download_results] forecast 2024-11-01 results files from arbutus.cloud-nowcast downloaded 2024-11-01 12:33:41,096 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 12:33:41,122 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:33:44,232 INFO [make_CHS_currents_file] running in process 391515 2024-11-01 12:33:44,233 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:33:44,234 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-11-01 12:33:44,235 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:34:31,919 INFO [make_CHS_currents_file] Made CHS currents file for 2024-11-01 for forecast 2024-11-01 12:34:31,923 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-11-01 12:34:31,949 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:34:33,102 INFO [update_forecast_datasets] running in process 391543 2024-11-01 12:34:33,103 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:34:33,104 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-11-01 12:34:33,105 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:34:33,267 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/01nov24 run 2024-11-01 12:34:33,368 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/01nov24 run: /results/SalishSea/rolling-forecasts/nemo 2024-11-01 12:34:33,369 INFO [update_forecast_datasets] nemo 2024-11-01 forecast rolling forecast datasets updated 2024-11-01 12:34:33,372 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-11-01 12:34:33,394 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-11-01 12:34:33,400 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:34:33,405 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:34:34,706 INFO [ping_erddap] running in process 391548 2024-11-01 12:34:34,707 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:34:34,707 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-11-01 12:34:34,708 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:34:34,756 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-11-01 12:34:34,758 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-01 12:34:36,126 INFO [make_surface_current_tiles] running in process 391551 2024-11-01 12:34:36,129 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:34:36,131 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2024-11-01 12:34:36,132 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:34:36,134 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-11-01 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/01nov24 2024-11-01 12:34:39,262 INFO [make_plots] running in process 391550 2024-11-01 12:34:39,263 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:34:39,264 INFO [make_plots] publishing log messages to tcp://*:5566 2024-11-01 12:34:39,265 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:34:39,922 INFO [make_plots] preparing render list for 2024-11-01 NEMO forecast publish figures 2024-11-01 12:34:40,041 INFO [make_plots] added Website_thumbnail figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,042 INFO [make_plots] added Threshold_website figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,043 INFO [make_plots] added PA_tidal_predictions figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,044 INFO [make_plots] added Boundary Bay figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,045 INFO [make_plots] added Campbell River figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,046 INFO [make_plots] added Cherry Point figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,047 INFO [make_plots] added Friday Harbor figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,047 INFO [make_plots] added Halfmoon Bay figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,048 INFO [make_plots] added Nanaimo figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,048 INFO [make_plots] added Neah Bay figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,049 INFO [make_plots] added New Westminster figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,050 INFO [make_plots] added Patricia Bay figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,051 INFO [make_plots] added Point Atkinson figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,051 INFO [make_plots] added Port Renfrew figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,052 INFO [make_plots] added Sand Heads figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,053 INFO [make_plots] added Sandy Cove figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,054 INFO [make_plots] added Squamish figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,054 INFO [make_plots] added Victoria figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,055 INFO [make_plots] added Woodwards Landing figure to 2024-11-01 NEMO forecast publish render list 2024-11-01 12:34:40,056 INFO [make_plots] starting to render nemo forecast publish 01nov24 figures 2024-11-01 12:37:21,764 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18022190 = 2024-11-01 14:32:40 UTC, 60.6% complete 2024-11-01 12:38:03,303 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 207, 31.9% complete 2024-11-01 12:38:29,799 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8043032 = 2024-11-01 15:00:40 UTC, 62.6% complete 2024-11-01 12:42:21,763 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18022377 = 2024-11-01 16:37:20 UTC, 69.3% complete 2024-11-01 12:43:03,304 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 315, 48.6% complete 2024-11-01 12:43:30,277 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8043240 = 2024-11-01 17:19:20 UTC, 72.2% complete 2024-11-01 12:46:04,362 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/01nov24 2024-11-01 12:47:21,780 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18022562 = 2024-11-01 18:40:40 UTC, 77.9% complete 2024-11-01 12:48:03,299 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 417, 64.4% complete 2024-11-01 12:48:30,722 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8043447 = 2024-11-01 19:37:20 UTC, 81.8% complete 2024-11-01 12:50:25,405 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-11-01 12:50:25,406 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-11-01 12:51:02,287 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:51:21,662 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/01nov24 2024-11-01 12:51:21,674 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-11-01 forecast into /results/nowcast-sys/figures/surface_currents/forecast/01nov24 2024-11-01 12:51:21,675 INFO [make_surface_current_tiles] surface current tile figures for 2024-11-01 forecast completed 2024-11-01 12:51:22,164 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-11-01 12:52:21,768 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18022746 = 2024-11-01 20:43:20 UTC, 86.4% complete 2024-11-01 12:52:53,000 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:52:53,185 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-11-01 12:52:54,126 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:53:03,300 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 525, 81.0% complete 2024-11-01 12:53:31,167 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8043654 = 2024-11-01 21:55:20 UTC, 91.4% complete 2024-11-01 12:54:05,911 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:54:42,467 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:55:19,432 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:55:55,268 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:56:30,670 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:57:06,953 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:57:21,759 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18022933 = 2024-11-01 22:48:00 UTC, 95.0% complete 2024-11-01 12:57:43,502 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:57:43,676 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-11-01 12:57:44,009 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:58:03,297 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 630, 97.2% complete 2024-11-01 12:58:21,590 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-10-27 00:00:00Z to 2024-11-03 11:55:00Z 2024-11-01 12:58:31,596 INFO [watch_NEMO_agrif] 01nov24nowcast-agrif on orcinus-nowcast-agrif: timestep: 8043839 = 2024-11-01 23:58:40 UTC, 100.0% complete 2024-11-01 12:58:57,952 INFO [make_plots] finished rendering nemo forecast publish 01nov24 figures 2024-11-01 12:58:57,952 INFO [make_plots] nemo publish plots for 2024-11-01 forecast completed 2024-11-01 12:58:57,958 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-11-01 12:58:58,127 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 12:59:03,676 INFO [make_feeds] running in process 393540 2024-11-01 12:59:03,678 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 12:59:03,679 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-11-01 12:59:03,680 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-11-01 12:59:33,049 INFO [make_feeds] ATOM feeds for 2024-11-01 forecast run completed 2024-11-01 12:59:33,052 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-11-01 13:02:21,758 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18023040 = 2024-11-01 23:59:20 UTC, 100.0% complete 2024-11-01 13:03:03,297 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-11-01 13:03:03,299 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-11-01 13:03:03,635 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:03:04,524 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 13:03:04,847 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-03-21'], host='localhost') 2024-11-01 13:03:04,854 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 13:03:04,859 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 13:03:04,991 INFO [download_wwatch3_results] running in process 393574 2024-11-01 13:03:04,992 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:03:04,992 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563 2024-11-01 13:03:04,993 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:03:06,190 INFO [download_results] running in process 393578 2024-11-01 13:03:06,191 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:03:06,192 INFO [download_results] publishing log messages to tcp://*:5569 2024-11-01 13:03:06,193 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:03:06,195 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21mar12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 13:03:10,418 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 13:03:31,821 INFO [watch_NEMO_agrif] NEMO AGRIF run 11410650.orca2.ibb on orcinus-nowcast-agrif completed 2024-11-01 13:03:31,826 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker 2024-11-01 13:03:32,154 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:03:34,078 INFO [download_results] running in process 393612 2024-11-01 13:03:34,079 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:03:34,081 INFO [download_results] publishing log messages to tcp://*:5565 2024-11-01 13:03:34,082 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:03:34,084 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/01nov24 to /results/SalishSea/nowcast-agrif.201702 2024-11-01 13:05:41,751 INFO [download_wwatch3_results] forecast 2024-11-01 results files from arbutus.cloud-nowcast downloaded 2024-11-01 13:05:41,755 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-11-01 13:05:41,929 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:05:44,145 INFO [update_forecast_datasets] running in process 393626 2024-11-01 13:05:44,148 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:05:44,152 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-11-01 13:05:44,155 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:05:44,158 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/01nov24 run 2024-11-01 13:05:44,163 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/01nov24 run 2024-11-01 13:05:44,331 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/01nov24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-11-01 13:05:44,334 INFO [update_forecast_datasets] wwatch3 2024-11-01 forecast rolling forecast datasets updated 2024-11-01 13:05:44,341 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-11-01 13:05:44,534 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-11-01 13:05:46,201 INFO [ping_erddap] running in process 393631 2024-11-01 13:05:46,205 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:05:46,208 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-11-01 13:05:46,213 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:05:46,220 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-11-01 13:05:46,227 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-01 13:05:46,404 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:05:56,070 INFO [make_plots] running in process 393636 2024-11-01 13:05:56,071 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:05:56,076 INFO [make_plots] publishing log messages to tcp://*:5563 2024-11-01 13:05:56,081 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:05:56,085 INFO [make_plots] preparing render list for 2024-11-01 WaveWatch3 forecast publish figures 2024-11-01 13:05:56,090 INFO [make_plots] added Halibut Bank figure to 2024-11-01 WaveWatch3 forecast publish render list 2024-11-01 13:05:56,094 INFO [make_plots] added Sentry Shoal figure to 2024-11-01 WaveWatch3 forecast publish render list 2024-11-01 13:05:56,097 INFO [make_plots] starting to render wwatch3 forecast publish 01nov24 figures 2024-11-01 13:05:58,049 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-11-01 13:06:09,882 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-11-01 13:06:16,994 INFO [make_plots] finished rendering wwatch3 forecast publish 01nov24 figures 2024-11-01 13:06:16,996 INFO [make_plots] wwatch3 publish plots for 2024-11-01 forecast completed 2024-11-01 13:06:17,003 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-11-01 13:07:22,100 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2024-11-01 13:07:22,106 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-11-01 13:07:22,467 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:07:26,635 INFO [download_results] running in process 393714 2024-11-01 13:07:26,638 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:07:26,642 INFO [download_results] publishing log messages to tcp://*:5563 2024-11-01 13:07:26,645 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:07:26,652 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/01nov24 to /results2/SalishSea/nowcast-green.202111 2024-11-01 13:11:37,361 INFO [download_results] hindcast 2012-03-21 results files from optimum-hindcast downloaded 2024-11-01 13:11:37,382 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 13:11:37,551 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-03-21'], host='localhost') 2024-11-01 13:11:39,168 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 13:19:13,089 ERROR [download_results] subprocess ['scp', '-pr', 'orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/01nov24', '/results/SalishSea/nowcast-agrif.201702'] failed with return code 1 2024-11-01 13:19:13,678 ERROR [download_results] b'kex_exchange_identification: read: Connection timed out' 2024-11-01 13:19:13,680 CRITICAL [download_results] nowcast-agrif 2024-11-01 results files download from orcinus-nowcast-agrif failed 2024-11-01 13:24:43,156 INFO [archive_tarball] hindcast *feb12 results files archived to robot.graham 2024-11-01 13:24:43,159 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-11-01 13:40:49,446 INFO [download_results] nowcast-green 2024-11-01 results files from arbutus.cloud-nowcast downloaded 2024-11-01 13:40:49,453 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 13:40:49,627 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:40:49,631 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2024-11-01 13:40:49,637 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:40:49,642 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:40:49,646 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 13:40:51,157 INFO [ping_erddap] running in process 393962 2024-11-01 13:40:51,160 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:40:51,161 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-11-01 13:40:51,162 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:40:51,319 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2024-11-01 13:40:51,327 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-01 13:40:53,130 INFO [make_averaged_dataset] running in process 393963 2024-11-01 13:40:53,131 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:40:53,132 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563 2024-11-01 13:40:53,133 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:40:53,135 INFO [make_averaged_dataset] creating day-averaged dataset for 01-Nov-2024 biology 2024-11-01 13:40:53,140 INFO [make_averaged_dataset] running in process 393964 2024-11-01 13:40:53,141 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:40:53,142 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564 2024-11-01 13:40:53,144 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:40:53,145 INFO [make_averaged_dataset] creating day-averaged dataset for 01-Nov-2024 chemistry 2024-11-01 13:40:53,147 INFO [make_averaged_dataset] running in process 393965 2024-11-01 13:40:53,148 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:40:53,149 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565 2024-11-01 13:40:53,150 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:40:53,151 INFO [make_averaged_dataset] creating day-averaged dataset for 01-Nov-2024 physics 2024-11-01 13:40:55,512 INFO [make_plots] running in process 393960 2024-11-01 13:40:55,513 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 13:40:55,514 INFO [make_plots] publishing log messages to tcp://*:5567 2024-11-01 13:40:55,515 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-11-01 13:40:56,313 INFO [make_plots] preparing render list for 2024-11-01 NEMO nowcast-green research figures 2024-11-01 13:40:58,050 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:40:58,766 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:40:59,475 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:00,176 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:00,899 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:01,620 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:02,345 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:03,160 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:03,995 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:04,811 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:05,634 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:06,375 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:07,098 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:07,099 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:07,100 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:07,101 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-11-01 NEMO nowcast-green research render list 2024-11-01 13:41:07,101 INFO [make_plots] starting to render nemo nowcast-green research 01nov24 figures 2024-11-01 13:41:35,524 INFO [make_averaged_dataset] day-averaged dataset for 01-Nov-2024 chemistry created 2024-11-01 13:41:35,527 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-11-01 13:42:08,720 INFO [make_averaged_dataset] day-averaged dataset for 01-Nov-2024 physics created 2024-11-01 13:42:08,724 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-11-01 13:42:20,823 INFO [make_averaged_dataset] day-averaged dataset for 01-Nov-2024 biology created 2024-11-01 13:42:20,826 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-11-01 14:48:17,611 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 14:48:18,032 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-03-26'], host='localhost') 2024-11-01 14:48:18,042 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 14:48:18,045 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 14:48:20,372 INFO [download_results] running in process 394561 2024-11-01 14:48:20,373 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 14:48:20,374 INFO [download_results] publishing log messages to tcp://*:5565 2024-11-01 14:48:20,376 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 14:48:20,377 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26mar12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 14:48:24,576 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 14:50:29,151 INFO [download_results] running in process 394726 2024-11-01 14:50:29,171 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 14:50:29,189 INFO [download_results] publishing log messages to tcp://*:5563 2024-11-01 14:50:29,207 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 14:50:29,210 CRITICAL [download_results] unrecognized host: orinus-nowcast-agrif 2024-11-01 14:50:30,085 CRITICAL [download_results] nowcast-agrif 2024-11-01 results files download from orinus-nowcast-agrif failed 2024-11-01 14:53:58,912 INFO [make_plots] finished rendering nemo nowcast-green research 01nov24 figures 2024-11-01 14:53:59,086 INFO [make_plots] nemo research plots for 2024-11-01 nowcast-green completed 2024-11-01 14:53:59,157 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-11-01 14:57:24,179 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20241101/18/ 2024-11-01 14:57:24,181 INFO [collect_weather] 2024-11-01 2.5km weather forecast 18 collection complete 2024-11-01 14:57:24,184 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-11-01 14:57:24,561 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2024-11-01 14:57:24,572 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2024-11-01 14:57:24,578 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2024-11-01 14:57:24,585 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-11-02'], host='localhost') 2024-11-01 14:57:24,763 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241101/18/ 2024-11-01 14:57:24,764 INFO [crop_gribs] 2024-11-01 18 GRIBs cropping complete 2024-11-01 14:57:24,767 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-11-01 14:57:28,521 INFO [download_weather] running in process 394799 2024-11-01 14:57:28,525 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 14:57:28,528 INFO [download_weather] publishing log messages to tcp://*:5562 2024-11-01 14:57:28,531 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-11-01 14:57:28,535 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20241101 2024-11-01 14:57:28,538 INFO [download_weather] running in process 394797 2024-11-01 14:57:28,542 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 14:57:28,546 INFO [download_weather] publishing log messages to tcp://*:5563 2024-11-01 14:57:28,555 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-11-01 14:57:28,559 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20241101 2024-11-01 14:57:28,562 INFO [collect_weather] running in process 394800 2024-11-01 14:57:28,572 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 14:57:28,575 INFO [collect_weather] publishing log messages to tcp://*:5566 2024-11-01 14:57:28,580 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-11-01 14:57:28,664 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2024-11-01 14:57:33,566 INFO [crop_gribs] running in process 394801 2024-11-01 14:57:33,570 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 14:57:33,573 INFO [crop_gribs] publishing log messages to tcp://*:5567 2024-11-01 14:57:33,577 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-11-01 14:57:33,580 INFO [crop_gribs] cropping 2024-11-02 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2024-11-01 14:57:33,587 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241102/00/ 2024-11-01 14:58:02,365 INFO [download_results] hindcast 2012-03-26 results files from optimum-hindcast downloaded 2024-11-01 14:58:02,386 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 14:58:02,588 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-03-26'], host='localhost') 2024-11-01 14:58:04,222 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 14:58:04,425 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2012-mar', 'robot.graham'], host='localhost') 2024-11-01 14:58:05,624 INFO [archive_tarball] running in process 394845 2024-11-01 14:58:05,625 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 14:58:05,627 INFO [archive_tarball] publishing log messages to tcp://*:5565 2024-11-01 14:58:05,628 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-11-01 14:58:05,629 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar12.tar from /ocean/sallen/timbucktwo/oxygen/*mar12/ 2024-11-01 14:59:49,031 INFO [download_results] running in process 394860 2024-11-01 14:59:49,033 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 14:59:49,034 INFO [download_results] publishing log messages to tcp://*:5568 2024-11-01 14:59:49,036 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 14:59:49,037 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/01nov24 to /results/SalishSea/nowcast-agrif.201702 2024-11-01 15:01:15,518 INFO [download_weather] 2024-11-01 1km weather forecast 00 downloads complete 2024-11-01 15:01:15,522 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-11-01 15:01:31,676 INFO [download_weather] 2024-11-01 1km weather forecast 12 downloads complete 2024-11-01 15:01:31,680 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-11-01 15:02:24,837 INFO [download_results] nowcast-agrif 2024-11-01 results files from orcinus-nowcast-agrif downloaded 2024-11-01 15:02:24,844 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 15:02:25,051 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-11-01'], host='localhost') 2024-11-01 15:02:32,867 INFO [make_plots] running in process 394895 2024-11-01 15:02:32,868 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 15:02:32,870 INFO [make_plots] publishing log messages to tcp://*:5562 2024-11-01 15:02:32,870 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-11-01 15:02:33,712 INFO [make_plots] preparing render list for 2024-11-01 NEMO nowcast-agrif research figures 2024-11-01 15:02:33,714 INFO [make_plots] added baynes_sound_surface figure to 2024-11-01 NEMO nowcast-agrif research render list 2024-11-01 15:02:33,715 INFO [make_plots] starting to render nemo nowcast-agrif research 01nov24 figures 2024-11-01 15:02:42,674 INFO [make_plots] finished rendering nemo nowcast-agrif research 01nov24 figures 2024-11-01 15:02:42,676 INFO [make_plots] nemo research plots for 2024-11-01 nowcast-agrif completed 2024-11-01 15:02:42,680 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-11-01 16:06:56,764 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar12.index from /ocean/dlatorne/oxygen-mar12.tar 2024-11-01 16:07:19,068 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-mar12.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-11-01 16:13:29,721 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 16:13:30,076 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-04-01'], host='localhost') 2024-11-01 16:13:30,084 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 16:13:30,090 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 16:13:31,994 INFO [download_results] running in process 395278 2024-11-01 16:13:31,995 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 16:13:31,996 INFO [download_results] publishing log messages to tcp://*:5564 2024-11-01 16:13:31,997 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 16:13:31,998 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01apr12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 16:13:36,141 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 16:23:15,363 INFO [download_results] hindcast 2012-04-01 results files from optimum-hindcast downloaded 2024-11-01 16:23:15,378 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 16:23:15,581 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-04-01'], host='localhost') 2024-11-01 16:23:17,517 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 17:43:41,515 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 17:43:41,934 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-04-06'], host='localhost') 2024-11-01 17:43:41,941 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 17:43:41,946 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 17:43:43,834 INFO [download_results] running in process 396247 2024-11-01 17:43:43,835 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 17:43:43,836 INFO [download_results] publishing log messages to tcp://*:5563 2024-11-01 17:43:43,837 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 17:43:43,838 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06apr12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 17:43:47,965 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 17:52:14,625 INFO [download_results] hindcast 2012-04-06 results files from optimum-hindcast downloaded 2024-11-01 17:52:14,641 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 17:52:14,847 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-04-06'], host='localhost') 2024-11-01 17:52:16,747 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 19:08:53,017 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 19:08:53,385 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-04-11'], host='localhost') 2024-11-01 19:08:53,392 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 19:08:53,399 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 19:08:55,304 INFO [download_results] running in process 396881 2024-11-01 19:08:55,305 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 19:08:55,306 INFO [download_results] publishing log messages to tcp://*:5562 2024-11-01 19:08:55,307 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 19:08:55,308 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11apr12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 19:09:00,831 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 19:17:36,155 INFO [download_results] hindcast 2012-04-11 results files from optimum-hindcast downloaded 2024-11-01 19:17:36,178 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 19:17:36,384 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-04-11'], host='localhost') 2024-11-01 19:17:38,287 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 20:27:02,525 INFO [archive_tarball] hindcast *mar12 results files archived to robot.graham 2024-11-01 20:27:02,528 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-11-01 20:39:04,643 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 20:39:05,015 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-04-16'], host='localhost') 2024-11-01 20:39:05,020 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 20:39:05,026 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 20:39:06,823 INFO [download_results] running in process 397604 2024-11-01 20:39:06,824 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 20:39:06,825 INFO [download_results] publishing log messages to tcp://*:5564 2024-11-01 20:39:06,826 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 20:39:06,827 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16apr12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 20:39:12,331 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 20:47:08,430 INFO [download_results] hindcast 2012-04-16 results files from optimum-hindcast downloaded 2024-11-01 20:47:08,450 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 20:47:08,658 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-04-16'], host='localhost') 2024-11-01 20:47:10,312 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 20:58:47,612 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20241102/00/ 2024-11-01 20:58:47,613 INFO [collect_weather] 2024-11-02 2.5km weather forecast 00 collection complete 2024-11-01 20:58:47,616 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-11-01 20:58:47,911 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241102/00/ 2024-11-01 20:58:47,912 INFO [crop_gribs] 2024-11-02 00 GRIBs cropping complete 2024-11-01 20:58:47,986 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2024-11-01 20:58:48,019 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-11-02'], host='localhost') 2024-11-01 20:58:48,036 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-11-01 20:58:49,462 INFO [collect_weather] running in process 397808 2024-11-01 20:58:49,464 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 20:58:49,465 INFO [collect_weather] publishing log messages to tcp://*:5563 2024-11-01 20:58:49,466 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-11-01 20:58:49,484 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2024-11-01 20:58:50,913 INFO [crop_gribs] running in process 397810 2024-11-01 20:58:50,914 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 20:58:50,915 INFO [crop_gribs] publishing log messages to tcp://*:5564 2024-11-01 20:58:50,916 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-11-01 20:58:50,917 INFO [crop_gribs] cropping 2024-11-02 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2024-11-01 20:58:50,931 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241102/06/ 2024-11-01 22:04:16,085 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 22:04:16,438 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-04-21'], host='localhost') 2024-11-01 22:04:16,445 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 22:04:16,450 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 22:04:17,772 INFO [download_results] running in process 398277 2024-11-01 22:04:17,774 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 22:04:17,775 INFO [download_results] publishing log messages to tcp://*:5562 2024-11-01 22:04:17,776 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 22:04:17,777 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21apr12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 22:04:22,107 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 22:12:15,788 INFO [download_results] hindcast 2012-04-21 results files from optimum-hindcast downloaded 2024-11-01 22:12:15,810 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 22:12:16,019 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-04-21'], host='localhost') 2024-11-01 22:12:17,902 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 23:29:26,930 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-01 23:29:27,291 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-04-26'], host='localhost') 2024-11-01 23:29:27,298 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 23:29:27,303 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-01 23:29:28,665 INFO [download_results] running in process 398638 2024-11-01 23:29:28,666 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 23:29:28,667 INFO [download_results] publishing log messages to tcp://*:5566 2024-11-01 23:29:28,668 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-01 23:29:28,669 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26apr12 to /ocean/sallen/timbucktwo/oxygen 2024-11-01 23:29:33,137 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-01 23:37:25,783 INFO [download_results] hindcast 2012-04-26 results files from optimum-hindcast downloaded 2024-11-01 23:37:25,805 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-01 23:37:26,010 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-04-26'], host='localhost') 2024-11-01 23:37:27,446 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-01 23:37:27,655 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2012-apr', 'robot.graham'], host='localhost') 2024-11-01 23:37:28,892 INFO [archive_tarball] running in process 398713 2024-11-01 23:37:28,894 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-01 23:37:28,895 INFO [archive_tarball] publishing log messages to tcp://*:5562 2024-11-01 23:37:28,896 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-11-01 23:37:28,897 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-apr12.tar from /ocean/sallen/timbucktwo/oxygen/*apr12/ 2024-11-02 00:45:04,064 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-apr12.index from /ocean/dlatorne/oxygen-apr12.tar 2024-11-02 00:45:25,241 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-apr12.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-11-02 00:59:38,047 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-02 00:59:38,404 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-05-01'], host='localhost') 2024-11-02 00:59:38,410 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-02 00:59:38,416 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-02 00:59:40,007 INFO [download_results] running in process 399271 2024-11-02 00:59:40,008 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 00:59:40,009 INFO [download_results] publishing log messages to tcp://*:5566 2024-11-02 00:59:40,011 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-02 00:59:40,012 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01may12 to /ocean/sallen/timbucktwo/oxygen 2024-11-02 00:59:44,446 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-02 01:08:07,666 INFO [download_results] hindcast 2012-05-01 results files from optimum-hindcast downloaded 2024-11-02 01:08:07,687 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-02 01:08:07,893 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-05-01'], host='localhost') 2024-11-02 01:08:09,810 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-02 02:24:48,983 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-02 02:24:49,333 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-05-06'], host='localhost') 2024-11-02 02:24:49,340 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-02 02:24:49,346 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-02 02:24:51,374 INFO [download_results] running in process 399778 2024-11-02 02:24:51,375 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:24:51,376 INFO [download_results] publishing log messages to tcp://*:5567 2024-11-02 02:24:51,378 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:24:51,379 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06may12 to /ocean/sallen/timbucktwo/oxygen 2024-11-02 02:24:56,002 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-02 02:33:17,037 INFO [download_results] hindcast 2012-05-06 results files from optimum-hindcast downloaded 2024-11-02 02:33:17,057 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-02 02:33:17,266 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-05-06'], host='localhost') 2024-11-02 02:33:19,088 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-02 02:56:39,195 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20241102/06/ 2024-11-02 02:56:39,196 INFO [collect_weather] 2024-11-02 2.5km weather forecast 06 collection complete 2024-11-02 02:56:39,200 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-11-02 02:56:39,574 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,581 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,586 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,592 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,598 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,603 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,608 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,613 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,621 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,626 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,646 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,650 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,675 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,690 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,706 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2024-11-02 02:56:39,715 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2024-11-02 02:56:39,728 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2024-11-02 02:56:39,740 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-11-01'], host='localhost') 2024-11-02 02:56:39,751 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2024-11-02 02:56:39,758 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2024-11-02 02:56:39,770 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2024-11-02 02:56:39,796 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20241102/06/ 2024-11-02 02:56:39,797 INFO [crop_gribs] 2024-11-02 06 GRIBs cropping complete 2024-11-02 02:56:39,799 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-11-02 02:56:40,433 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2024-11-02 02:56:42,832 INFO [collect_weather] running in process 399998 2024-11-02 02:56:42,834 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:42,835 INFO [collect_weather] publishing log messages to tcp://*:5563 2024-11-02 02:56:42,835 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:42,849 INFO [collect_NeahBay_ssh] running in process 399997 2024-11-02 02:56:42,850 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:42,851 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564 2024-11-02 02:56:42,852 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:42,852 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-11-02 2024-11-02 02:56:42,908 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2024-11-02 02:56:43,831 INFO [collect_river_data] running in process 399977 2024-11-02 02:56:43,832 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:43,835 INFO [collect_river_data] publishing log messages to tcp://*:5565 2024-11-02 02:56:43,836 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:43,838 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-11-01 2024-11-02 02:56:43,858 INFO [collect_river_data] running in process 399979 2024-11-02 02:56:43,859 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:43,860 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-11-02 02:56:43,863 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:43,864 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-11-01 2024-11-02 02:56:43,865 INFO [collect_river_data] running in process 399987 2024-11-02 02:56:43,866 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:43,867 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-11-02 02:56:43,868 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:43,869 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-11-01 2024-11-02 02:56:43,883 INFO [collect_river_data] running in process 399990 2024-11-02 02:56:43,886 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:43,888 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-11-02 02:56:43,891 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:43,892 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-11-01 2024-11-02 02:56:43,900 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-11-01 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2024-11-02 02:56:43,901 INFO [collect_river_data] ECCC Capilano river data collection for 2024-11-01 completed 2024-11-02 02:56:43,903 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:43,923 INFO [collect_river_data] running in process 399991 2024-11-02 02:56:43,923 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:43,924 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-11-02 02:56:43,925 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:43,926 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-11-01 2024-11-02 02:56:43,943 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2024-11-02 02:56:43,945 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2024-11-02 02:56:43,946 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2024-11-02 02:56:43,947 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-11-01 completed 2024-11-02 02:56:43,948 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-11-01 completed 2024-11-02 02:56:43,949 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-11-01 completed 2024-11-02 02:56:43,968 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2024-11-02 02:56:43,968 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-11-01 completed 2024-11-02 02:56:43,989 INFO [collect_river_data] running in process 399981 2024-11-02 02:56:43,990 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:43,992 INFO [collect_river_data] publishing log messages to tcp://*:5571 2024-11-02 02:56:43,993 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:43,994 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-11-01 2024-11-02 02:56:43,997 INFO [collect_river_data] running in process 399980 2024-11-02 02:56:43,999 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,002 INFO [collect_river_data] publishing log messages to tcp://*:5572 2024-11-02 02:56:44,004 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,004 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-11-01 2024-11-02 02:56:44,016 INFO [collect_river_data] running in process 399986 2024-11-02 02:56:44,018 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,020 INFO [collect_river_data] publishing log messages to tcp://*:5573 2024-11-02 02:56:44,021 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,022 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-11-01 2024-11-02 02:56:44,025 INFO [collect_river_data] running in process 399988 2024-11-02 02:56:44,028 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,029 INFO [collect_river_data] publishing log messages to tcp://*:5574 2024-11-02 02:56:44,029 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,030 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-11-01 2024-11-02 02:56:44,032 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-11-01 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2024-11-02 02:56:44,033 INFO [collect_river_data] ECCC Englishman river data collection for 2024-11-01 completed 2024-11-02 02:56:44,042 INFO [collect_river_data] running in process 399984 2024-11-02 02:56:44,046 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,048 INFO [collect_river_data] publishing log messages to tcp://*:5575 2024-11-02 02:56:44,049 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,051 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-11-01 2024-11-02 02:56:44,052 INFO [collect_river_data] running in process 399982 2024-11-02 02:56:44,055 INFO [collect_river_data] running in process 399983 2024-11-02 02:56:44,055 INFO [collect_river_data] running in process 399985 2024-11-02 02:56:44,056 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,057 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,058 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,058 INFO [collect_river_data] publishing log messages to tcp://*:5576 2024-11-02 02:56:44,060 INFO [collect_river_data] publishing log messages to tcp://*:5577 2024-11-02 02:56:44,060 INFO [collect_river_data] publishing log messages to tcp://*:5578 2024-11-02 02:56:44,061 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,062 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,062 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,063 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-11-01 2024-11-02 02:56:44,063 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-11-01 2024-11-02 02:56:44,064 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-11-01 2024-11-02 02:56:44,065 INFO [collect_river_data] running in process 399989 2024-11-02 02:56:44,067 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,068 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2024-11-02 02:56:44,069 INFO [collect_river_data] publishing log messages to tcp://*:5579 2024-11-02 02:56:44,070 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2024-11-02 02:56:44,070 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-11-01 completed 2024-11-02 02:56:44,071 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,072 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-11-01 completed 2024-11-02 02:56:44,073 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-11-01 2024-11-02 02:56:44,086 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-11-01 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2024-11-02 02:56:44,087 INFO [collect_river_data] ECCC Fraser river data collection for 2024-11-01 completed 2024-11-02 02:56:44,093 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2024-11-02 02:56:44,093 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-11-01 completed 2024-11-02 02:56:44,096 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2024-11-02 02:56:44,097 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/RobertsCreek_flow 2024-11-02 02:56:44,097 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2024-11-02 02:56:44,098 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-11-01 completed 2024-11-02 02:56:44,099 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-11-01 completed 2024-11-02 02:56:44,100 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-11-01 completed 2024-11-02 02:56:44,101 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-11-01 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2024-11-02 02:56:44,103 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-11-01 completed 2024-11-02 02:56:44,277 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:44,304 INFO [collect_NeahBay_ssh] 2024-11-02T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2024-11-02 02:56:44,457 INFO [get_vfpa_hadcp] running in process 399996 2024-11-02 02:56:44,458 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,459 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5590 2024-11-02 02:56:44,460 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,460 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-11-01 2024-11-02 02:56:44,462 INFO [grib_to_netcdf] running in process 400055 2024-11-02 02:56:44,463 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,464 INFO [grib_to_netcdf] publishing log messages to tcp://*:5591 2024-11-02 02:56:44,464 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,465 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-11-02 forecast2 run 2024-11-02 02:56:44,476 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:44,537 INFO [crop_gribs] running in process 399999 2024-11-02 02:56:44,538 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,539 INFO [crop_gribs] publishing log messages to tcp://*:5565 2024-11-02 02:56:44,540 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,540 INFO [crop_gribs] cropping 2024-11-02 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2024-11-02 02:56:44,550 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20241102/12/ 2024-11-02 02:56:44,724 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:44,843 INFO [get_onc_ctd] running in process 399993 2024-11-02 02:56:44,846 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,849 INFO [get_onc_ctd] publishing log messages to tcp://*:5568 2024-11-02 02:56:44,851 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,854 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-11-01 2024-11-02 02:56:44,881 INFO [get_onc_ferry] running in process 399994 2024-11-02 02:56:44,882 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,884 INFO [get_onc_ferry] publishing log messages to tcp://*:5592 2024-11-02 02:56:44,885 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,886 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-11-01 2024-11-02 02:56:44,904 INFO [get_onc_ctd] running in process 399992 2024-11-02 02:56:44,905 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:44,906 INFO [get_onc_ctd] publishing log messages to tcp://*:5593 2024-11-02 02:56:44,907 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:44,908 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-11-01 2024-11-02 02:56:44,925 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:45,163 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:45,383 INFO [get_vfpa_hadcp] created /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202411.nc 2024-11-02 02:56:45,399 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:45,589 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:45,781 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:45,970 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:46,162 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:46,350 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:46,516 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-11-01 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202411.nc 2024-11-02 02:56:46,517 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-11 netcdf file 2024-11-02 02:56:46,542 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:46,725 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-11-02 02:56:46,909 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-11-02 02:56:47,091 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-11-02'], host='localhost') 2024-11-02 02:56:47,096 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2024-11-02 02:56:47,320 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2024-11-02 02:56:48,673 INFO [ping_erddap] running in process 400452 2024-11-02 02:56:48,674 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:48,675 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-11-02 02:56:48,676 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:48,678 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2024-11-02 02:56:48,680 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-02 02:56:49,486 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-11-01 2024-11-02 02:56:49,964 CRITICAL [get_onc_ctd] 2024-11-01 ONC SCVIP CTD T&S file creation failed 2024-11-02 02:56:50,334 INFO [make_ssh_files] running in process 400450 2024-11-02 02:56:50,335 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:50,336 INFO [make_ssh_files] publishing log messages to tcp://*:5564 2024-11-02 02:56:50,337 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:50,338 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-11-02 NOAA Neah Bay 00Z observation and forecast values 2024-11-02 02:56:53,904 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m11d03.nc 2024-11-02 02:56:55,237 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d28.nc 2024-11-02 02:56:55,418 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d29.nc 2024-11-02 02:56:55,594 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d30.nc 2024-11-02 02:56:55,793 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m10d31.nc 2024-11-02 02:56:56,016 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d01.nc 2024-11-02 02:56:56,212 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d02.nc 2024-11-02 02:56:56,286 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m11d04.nc 2024-11-02 02:56:56,292 INFO [grib_to_netcdf] 2024-11-02 NEMO-atmos forcing file for forecast2 created 2024-11-02 02:56:56,294 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-11-02 02:56:56,402 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d03.nc 2024-11-02 02:56:56,621 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d04.nc 2024-11-02 02:56:56,816 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m11d05.nc 2024-11-02 02:56:56,818 CRITICAL [make_ssh_files] /results/forcing/sshNeahBay/obs/ssh_y2024m11d01.nc was not created; using ../fcst/ssh_y2024m11d01.nc instead via symlink 2024-11-02 02:56:57,312 INFO [make_ssh_files] sea surface height boundary file for 2024-11-02 forecast2 run created 2024-11-02 02:56:57,315 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-11-02 02:56:57,495 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-11-02 02:56:57,502 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-11-02 02:56:58,848 INFO [make_runoff_file] running in process 400510 2024-11-02 02:56:58,849 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:58,850 INFO [make_runoff_file] publishing log messages to tcp://*:5564 2024-11-02 02:56:58,850 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:58,871 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-11-02 02:56:59,011 INFO [make_runoff_file] 2024-11-02 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-11-02 02:56:59,013 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-11-02 02:56:59,422 INFO [make_v202111_runoff_file] running in process 400509 2024-11-02 02:56:59,424 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:56:59,425 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5567 2024-11-02 02:56:59,426 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:56:59,427 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-11-01 2024-11-02 02:57:00,376 ERROR [make_v202111_runoff_file] no 2024-11-01 discharge obs for Skagit_MountVernon: patching 2024-11-02 02:57:00,908 ERROR [make_v202111_runoff_file] no 2024-11-01 discharge obs for Snohomish_Monroe: patching 2024-11-02 02:57:01,117 ERROR [make_v202111_runoff_file] no 2024-11-01 discharge obs for Nisqually_McKenna: patching 2024-11-02 02:57:01,204 ERROR [make_v202111_runoff_file] no 2024-11-01 discharge obs for Greenwater_Greenwater: patching 2024-11-02 02:57:02,544 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-11-01: /results/forcing/rivers/R202108Dailies_y2024m11d01.nc 2024-11-02 02:57:02,545 INFO [make_v202111_runoff_file] 2024-11-01 runoff file creation completed 2024-11-02 02:57:02,547 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-11-02 02:57:02,730 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2024-11-02 02:57:02,733 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2024-11-02 02:57:02,737 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2024-11-02 02:57:02,743 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2024-11-02 02:57:03,962 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-11-01 2024-11-02 02:57:04,121 INFO [upload_forcing] running in process 400551 2024-11-02 02:57:04,122 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:57:04,123 INFO [upload_forcing] publishing log messages to tcp://*:5564 2024-11-02 02:57:04,124 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:57:04,138 INFO [upload_forcing] running in process 400552 2024-11-02 02:57:04,139 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:57:04,140 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-11-02 02:57:04,141 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:57:04,147 INFO [upload_forcing] running in process 400550 2024-11-02 02:57:04,148 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:57:04,149 INFO [upload_forcing] publishing log messages to tcp://*:5569 2024-11-02 02:57:04,150 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:57:04,218 INFO [upload_forcing] running in process 400549 2024-11-02 02:57:04,219 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:57:04,220 INFO [upload_forcing] publishing log messages to tcp://*:5570 2024-11-02 02:57:04,222 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:57:05,442 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m11d01.nc 2024-11-02 02:57:05,494 INFO [upload_forcing] forecast2 2024-11-02 forcing files upload to optimum-hindcast completed 2024-11-02 02:57:05,499 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-02 02:57:06,819 INFO [upload_forcing] forecast2 2024-11-02 forcing files upload to arbutus.cloud-nowcast completed 2024-11-02 02:57:06,824 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-02 02:57:07,030 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-11-02'], host='localhost') 2024-11-02 02:57:08,010 WARNING [get_onc_ctd] no SEVIP salinity data 2024-11-02 02:57:08,289 INFO [get_onc_ctd] 2024-11-01 ONC SEVIP CTD T&S file created 2024-11-02 02:57:08,292 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2024-11-02 02:57:08,503 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2024-11-02 02:57:08,556 INFO [make_forcing_links] running in process 400577 2024-11-02 02:57:08,557 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:57:08,559 INFO [make_forcing_links] publishing log messages to tcp://*:5567 2024-11-02 02:57:08,560 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:57:09,656 INFO [ping_erddap] running in process 400582 2024-11-02 02:57:09,658 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:57:09,659 INFO [ping_erddap] publishing log messages to tcp://*:5568 2024-11-02 02:57:09,660 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:57:09,662 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2024-11-02 02:57:09,665 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-02 02:57:09,727 INFO [make_forcing_links] forecast2 2024-11-02 forcing file links on arbutus.cloud-nowcast created 2024-11-02 02:57:09,863 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-11-02 02:57:10,065 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-11-02'], host='arbutus.cloud-nowcast') 2024-11-02 02:57:11,098 INFO [upload_forcing] forecast2 2024-11-02 forcing files upload to robot.graham completed 2024-11-02 02:57:11,102 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-11-02 02:57:12,194 INFO [run_NEMO] running in process 24812 2024-11-02 02:57:12,195 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:57:12,197 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-11-02 02:57:12,209 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:57:13,148 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/01nov24forecast2_2024-11-02T025712.278500-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-11-02 02:57:13,169 INFO [run_NEMO] forecast2 NEMO run for 2024-11-02 on arbutus.cloud-nowcast started 2024-11-02 02:57:13,173 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-11-02 02:57:13,386 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-11-02 02:57:19,481 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-11-02 02:57:19,913 ERROR [upload_forcing] Traceback (most recent call last): 2024-11-02 02:57:19,914 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2327, in _check_banner 2024-11-02 02:57:19,916 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-11-02 02:57:19,917 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-02 02:57:19,919 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 381, in readline 2024-11-02 02:57:19,921 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-11-02 02:57:19,923 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-11-02 02:57:19,924 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/packet.py", line 626, in _read_timeout 2024-11-02 02:57:19,925 ERROR [upload_forcing] raise socket.timeout() 2024-11-02 02:57:19,927 ERROR [upload_forcing] TimeoutError 2024-11-02 02:57:19,929 ERROR [upload_forcing] 2024-11-02 02:57:19,930 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-11-02 02:57:19,932 ERROR [upload_forcing] 2024-11-02 02:57:19,934 ERROR [upload_forcing] Traceback (most recent call last): 2024-11-02 02:57:19,935 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2143, in run 2024-11-02 02:57:19,937 ERROR [upload_forcing] self._check_banner() 2024-11-02 02:57:19,939 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2331, in _check_banner 2024-11-02 02:57:19,941 ERROR [upload_forcing] raise SSHException( 2024-11-02 02:57:19,943 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-11-02 02:57:19,944 ERROR [upload_forcing] 2024-11-02 02:57:19,957 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 485, in connect self._auth( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 818, in _auth raise saved_exception File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 730, in _auth key = self._key_from_filepath( ^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 638, in _key_from_filepath key = klass.from_private_key_file(key_path, password) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/pkey.py", line 421, in from_private_key_file key = cls(filename=filename, password=password) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/ed25519key.py", line 60, in __init__ pkformat, data = self._read_private_key("OPENSSH", f) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/pkey.py", line 529, in _read_private_key raise SSHException( paramiko.ssh_exception.SSHException: encountered RSA key, expected OPENSSH key During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 94, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 85, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/client.py", line 451, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 738, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2143, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/paramiko/transport.py", line 2331, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-11-02 02:57:20,463 INFO [watch_NEMO] running in process 25071 2024-11-02 02:57:20,464 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 02:57:20,466 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-11-02 02:57:20,467 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-11-02 02:57:20,540 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-11-02 02:57:29,974 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-11-01 2024-11-02 02:57:54,516 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-11-01 2024-11-02 02:58:18,979 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-11-01 2024-11-02 02:58:35,473 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-11-01 2024-11-02 02:58:52,896 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-11-01 2024-11-02 02:59:01,163 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-11-01 2024-11-02 02:59:10,291 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-11-01; substituting empty dataset 2024-11-02 02:59:10,291 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-11-01 2024-11-02 02:59:18,367 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-11-01; substituting empty dataset 2024-11-02 02:59:18,759 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-11-01 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-11-02 02:59:18,775 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-11-01 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-11-02 03:01:14,569 INFO [get_onc_ferry] 2024-11-01 ONC TWDP ferry data file created 2024-11-02 03:01:14,571 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker 2024-11-02 03:01:14,782 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2024-11-02 03:01:16,120 INFO [ping_erddap] running in process 400611 2024-11-02 03:01:16,121 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:01:16,122 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-11-02 03:01:16,124 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:01:16,125 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2024-11-02 03:01:16,127 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-02 03:02:20,542 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18025638 = 2024-11-03 04:51:20 UTC, 16.2% complete 2024-11-02 03:07:20,543 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18026108 = 2024-11-03 10:04:40 UTC, 33.6% complete 2024-11-02 03:12:20,544 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18026587 = 2024-11-03 15:24:00 UTC, 51.4% complete 2024-11-02 03:17:20,546 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18027096 = 2024-11-03 21:03:20 UTC, 70.2% complete 2024-11-02 03:22:20,547 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18027592 = 2024-11-04 02:34:00 UTC, 88.6% complete 2024-11-02 03:27:20,551 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2024-11-02 03:27:20,555 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-11-02 03:27:20,991 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-11-02'], host='arbutus.cloud-nowcast') 2024-11-02 03:27:20,998 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-11-02'], host='arbutus.cloud-nowcast') 2024-11-02 03:27:21,002 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-11-01'], host='localhost') 2024-11-02 03:27:22,562 INFO [download_results] running in process 400705 2024-11-02 03:27:22,563 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:27:22,564 INFO [download_results] publishing log messages to tcp://*:5564 2024-11-02 03:27:22,565 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:27:22,567 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/01nov24 to /results/SalishSea/forecast2.202111 2024-11-02 03:27:33,912 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-11-02 forecast2 run 2024-11-02 03:27:33,916 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-11-02 03:29:35,104 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-11-02 forecast2 run 2024-11-02 03:29:35,109 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-11-02 03:29:35,329 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-11-02'], host='arbutus.cloud-nowcast') 2024-11-02 03:29:37,774 INFO [run_ww3] running in process 26903 2024-11-02 03:29:37,776 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:29:37,777 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-11-02 03:29:37,778 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:29:37,782 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2024-11-02T032937.728132-0700 2024-11-02 03:29:37,822 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2024-11-02T032937.728132-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-11-02 03:29:37,824 INFO [run_ww3] forecast2 WaveWatch3 run for 2024-11-02 on arbutus.cloud-nowcast started 2024-11-02 03:29:37,921 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-11-02 03:29:38,138 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-11-02 03:29:40,935 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-11-02 03:34:40,929 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 99, 18.3% complete 2024-11-02 03:39:40,924 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete 2024-11-02 03:44:40,932 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 309, 57.2% complete 2024-11-02 03:49:40,948 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 423, 78.3% complete 2024-11-02 03:52:11,457 INFO [download_results] forecast2 2024-11-01 results files from arbutus.cloud-nowcast downloaded 2024-11-02 03:52:11,462 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-02 03:52:11,670 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-11-01'], host='localhost') 2024-11-02 03:52:14,149 INFO [make_CHS_currents_file] running in process 400834 2024-11-02 03:52:14,150 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:52:14,151 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564 2024-11-02 03:52:14,152 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:52:50,218 INFO [make_CHS_currents_file] Made CHS currents file for 2024-11-01 for forecast2 2024-11-02 03:52:50,221 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-11-02 03:52:50,430 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-11-01'], host='localhost') 2024-11-02 03:52:51,570 INFO [update_forecast_datasets] running in process 400859 2024-11-02 03:52:51,572 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:52:51,573 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2024-11-02 03:52:51,574 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:52:51,575 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/01nov24 run 2024-11-02 03:54:40,900 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 525, 97.2% complete 2024-11-02 03:55:00,996 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-11-02 03:55:01,351 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2012-05-11'], host='localhost') 2024-11-02 03:55:01,357 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-02 03:55:01,361 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-11-02 03:55:02,724 INFO [download_results] running in process 400888 2024-11-02 03:55:02,725 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:55:02,726 INFO [download_results] publishing log messages to tcp://*:5568 2024-11-02 03:55:02,727 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:55:02,728 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11may12 to /ocean/sallen/timbucktwo/oxygen 2024-11-02 03:55:07,016 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-11-02 03:57:25,868 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/01nov24 run: /results/SalishSea/rolling-forecasts/nemo 2024-11-02 03:57:25,869 INFO [update_forecast_datasets] nemo 2024-11-01 forecast2 rolling forecast datasets updated 2024-11-02 03:57:25,872 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-11-02 03:57:26,095 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-11-02 03:57:26,102 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-11-01'], host='localhost') 2024-11-02 03:57:26,108 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-11-01'], host='localhost') 2024-11-02 03:57:28,054 INFO [ping_erddap] running in process 400970 2024-11-02 03:57:28,056 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:57:28,057 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-11-02 03:57:28,058 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:57:28,228 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-11-02 03:57:28,230 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-02 03:57:29,598 INFO [make_surface_current_tiles] running in process 400973 2024-11-02 03:57:29,599 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:57:29,600 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564 2024-11-02 03:57:29,602 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:57:29,604 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-11-01 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/01nov24 2024-11-02 03:57:33,843 INFO [make_plots] running in process 400972 2024-11-02 03:57:33,844 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:57:33,845 INFO [make_plots] publishing log messages to tcp://*:5566 2024-11-02 03:57:33,847 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-11-02 03:57:34,777 INFO [make_plots] preparing render list for 2024-11-01 NEMO forecast2 publish figures 2024-11-02 03:57:34,940 INFO [make_plots] added Website_thumbnail figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,943 INFO [make_plots] added Threshold_website figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,946 INFO [make_plots] added PA_tidal_predictions figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,949 INFO [make_plots] added Boundary Bay figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,950 INFO [make_plots] added Campbell River figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,951 INFO [make_plots] added Cherry Point figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,954 INFO [make_plots] added Friday Harbor figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,959 INFO [make_plots] added Halfmoon Bay figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,963 INFO [make_plots] added Nanaimo figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,965 INFO [make_plots] added Neah Bay figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,966 INFO [make_plots] added New Westminster figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,967 INFO [make_plots] added Patricia Bay figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,968 INFO [make_plots] added Point Atkinson figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,968 INFO [make_plots] added Port Renfrew figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,969 INFO [make_plots] added Sand Heads figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,970 INFO [make_plots] added Sandy Cove figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,972 INFO [make_plots] added Squamish figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,975 INFO [make_plots] added Victoria figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,978 INFO [make_plots] added Woodwards Landing figure to 2024-11-01 NEMO forecast2 publish render list 2024-11-02 03:57:34,981 INFO [make_plots] starting to render nemo forecast2 publish 01nov24 figures 2024-11-02 03:59:40,901 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2024-11-02 03:59:40,905 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-11-02 03:59:41,508 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-11-02'], host='localhost') 2024-11-02 03:59:43,669 INFO [download_wwatch3_results] running in process 401421 2024-11-02 03:59:43,670 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 03:59:43,672 INFO [download_wwatch3_results] publishing log messages to tcp://*:5569 2024-11-02 03:59:43,674 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-11-02 04:02:01,920 INFO [download_wwatch3_results] forecast2 2024-11-02 results files from arbutus.cloud-nowcast downloaded 2024-11-02 04:02:01,924 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-11-02 04:02:02,342 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-11-02'], host='localhost') 2024-11-02 04:02:04,457 INFO [update_forecast_datasets] running in process 401835 2024-11-02 04:02:04,458 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 04:02:04,459 INFO [update_forecast_datasets] publishing log messages to tcp://*:5569 2024-11-02 04:02:04,460 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-11-02 04:02:04,461 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/02nov24 run 2024-11-02 04:02:04,463 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/02nov24 run 2024-11-02 04:02:41,210 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/02nov24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-11-02 04:02:41,222 INFO [update_forecast_datasets] wwatch3 2024-11-02 forecast2 rolling forecast datasets updated 2024-11-02 04:02:41,225 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-11-02 04:02:41,643 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-11-02 04:02:43,481 INFO [ping_erddap] running in process 401954 2024-11-02 04:02:43,483 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 04:02:43,484 INFO [ping_erddap] publishing log messages to tcp://*:5569 2024-11-02 04:02:43,485 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-11-02 04:02:43,487 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-11-02 04:02:43,489 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-11-02 04:02:43,897 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-11-02'], host='localhost') 2024-11-02 04:02:52,283 INFO [make_plots] running in process 401962 2024-11-02 04:02:52,285 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 04:02:52,286 INFO [make_plots] publishing log messages to tcp://*:5569 2024-11-02 04:02:52,287 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-11-02 04:02:52,288 INFO [make_plots] preparing render list for 2024-11-02 WaveWatch3 forecast2 publish figures 2024-11-02 04:02:52,289 INFO [make_plots] added Halibut Bank figure to 2024-11-02 WaveWatch3 forecast2 publish render list 2024-11-02 04:02:52,290 INFO [make_plots] added Sentry Shoal figure to 2024-11-02 WaveWatch3 forecast2 publish render list 2024-11-02 04:02:52,291 INFO [make_plots] starting to render wwatch3 forecast2 publish 02nov24 figures 2024-11-02 04:02:53,424 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-11-02 04:03:04,928 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-11-02 04:03:12,500 INFO [make_plots] finished rendering wwatch3 forecast2 publish 02nov24 figures 2024-11-02 04:03:12,501 INFO [make_plots] wwatch3 publish plots for 2024-11-02 forecast2 completed 2024-11-02 04:03:12,504 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-11-02 04:03:59,724 INFO [download_results] hindcast 2012-05-11 results files from optimum-hindcast downloaded 2024-11-02 04:03:59,749 INFO [manager] checklist updated with [results files] items from download_results worker 2024-11-02 04:04:00,104 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2012-05-11'], host='localhost') 2024-11-02 04:04:01,872 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-11-02 04:09:26,777 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/01nov24 2024-11-02 04:12:27,399 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-11-02 04:12:27,400 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-11-02 04:13:04,501 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:14:14,293 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/01nov24 2024-11-02 04:14:14,302 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-11-01 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/01nov24 2024-11-02 04:14:14,304 INFO [make_surface_current_tiles] surface current tile figures for 2024-11-01 forecast2 completed 2024-11-02 04:14:14,580 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-11-02 04:14:55,847 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:14:56,032 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-11-02 04:14:56,855 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:16:08,731 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:16:44,995 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:17:21,513 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:17:58,115 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:18:34,251 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:19:11,147 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:19:47,895 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:19:48,069 ERROR [make_plots] unexpected TypeError in make_figure: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure fig = fig_func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 118, in make_figure plot_data = _prep_plot_data( ^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/publish/compare_tide_prediction_max_ssh.py", line 149, in _prep_plot_data data_tools.get_chs_tides( File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in get_chs_tides water_levels.extend(event["value"] for event in response.json()) File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 496, in water_levels.extend(event["value"] for event in response.json()) ~~~~~^^^^^^^^^ TypeError: string indices must be integers, not 'str' 2024-11-02 04:19:48,381 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:20:25,224 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-10-28 00:00:00Z to 2024-11-04 05:55:00Z 2024-11-02 04:21:02,131 INFO [make_plots] finished rendering nemo forecast2 publish 01nov24 figures 2024-11-02 04:21:02,132 INFO [make_plots] nemo publish plots for 2024-11-01 forecast2 completed 2024-11-02 04:21:02,140 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-11-02 04:21:02,478 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-11-01'], host='localhost') 2024-11-02 04:21:05,991 INFO [make_feeds] running in process 403282 2024-11-02 04:21:05,992 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 04:21:05,993 INFO [make_feeds] publishing log messages to tcp://*:5564 2024-11-02 04:21:05,995 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-11-02 04:21:36,012 INFO [make_feeds] ATOM feeds for 2024-11-01 forecast2 run completed 2024-11-02 04:21:36,016 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-11-02 04:21:36,310 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2024-11-02 04:21:37,472 INFO [clear_checklist] running in process 403305 2024-11-02 04:21:37,473 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 04:21:37,474 INFO [clear_checklist] publishing log messages to tcp://*:5564 2024-11-02 04:21:37,475 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2024-11-02 04:21:37,476 INFO [clear_checklist] requesting that manager clear system state checklist 2024-11-02 04:21:37,479 INFO [manager] writing checklist to log file 2024-11-02 04:21:37,526 INFO [manager] checklist cleared 2024-11-02 04:21:37,527 INFO [clear_checklist] nowcast system checklist cleared 2024-11-02 04:21:37,529 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2024-11-02 04:21:38,713 INFO [rotate_logs] running in process 403310 2024-11-02 04:21:38,714 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-11-02 04:21:38,716 INFO [rotate_logs] publishing log messages to tcp://*:5564 2024-11-02 04:21:38,717 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2024-11-02 04:21:38,718 INFO [rotate_logs] rotating log files