2024-03-23 04:45:24,487 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2024-03-23 04:45:24,493 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2024-03-23 04:45:24,498 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2024-03-23 04:45:24,500 INFO [rotate_logs] log files rotated 2024-03-23 04:45:24,505 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2024-03-23 04:54:52,354 INFO [archive_tarball] hindcast *jun19 results files archived to robot.graham 2024-03-23 04:54:52,359 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-03-23 05:24:25,194 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 05:24:25,386 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-01'], host='localhost') 2024-03-23 05:24:25,393 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 05:24:25,398 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 05:24:26,447 INFO [download_results] running in process 1563437 2024-03-23 05:24:26,448 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 05:24:26,449 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 05:24:26,451 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 05:24:26,452 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/01jul19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 05:24:30,606 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 05:24:55,188 INFO [download_results] hindcast 2019-07-01 results files from optimum-hindcast downloaded 2024-03-23 05:24:55,195 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 05:24:55,202 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-01'], host='localhost') 2024-03-23 05:24:56,124 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 06:14:31,292 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 06:14:31,489 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-06'], host='localhost') 2024-03-23 06:14:31,495 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 06:14:31,499 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 06:14:32,477 INFO [download_results] running in process 1564048 2024-03-23 06:14:32,478 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 06:14:32,480 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 06:14:32,481 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 06:14:32,482 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/06jul19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 06:14:36,649 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 06:15:01,332 INFO [download_results] hindcast 2019-07-06 results files from optimum-hindcast downloaded 2024-03-23 06:15:01,339 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 06:15:01,345 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-06'], host='localhost') 2024-03-23 06:15:02,173 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 07:04:37,352 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 07:04:37,546 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-11'], host='localhost') 2024-03-23 07:04:37,553 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 07:04:37,557 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 07:04:38,536 INFO [download_results] running in process 1564241 2024-03-23 07:04:38,537 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 07:04:38,539 INFO [download_results] publishing log messages to tcp://*:5565 2024-03-23 07:04:38,540 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 07:04:38,541 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/11jul19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 07:04:42,604 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 07:05:07,525 INFO [download_results] hindcast 2019-07-11 results files from optimum-hindcast downloaded 2024-03-23 07:05:07,533 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 07:05:07,541 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-11'], host='localhost') 2024-03-23 07:05:08,411 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 07:54:43,458 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 07:54:43,653 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-16'], host='localhost') 2024-03-23 07:54:43,660 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 07:54:43,664 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 07:54:44,671 INFO [download_results] running in process 1564481 2024-03-23 07:54:44,672 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 07:54:44,673 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 07:54:44,674 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 07:54:44,675 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/16jul19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 07:54:48,739 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 07:55:13,740 INFO [download_results] hindcast 2019-07-16 results files from optimum-hindcast downloaded 2024-03-23 07:55:13,747 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 07:55:13,754 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-16'], host='localhost') 2024-03-23 07:55:14,613 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 08:44:49,531 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 08:44:49,742 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-21'], host='localhost') 2024-03-23 08:44:49,749 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 08:44:49,753 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 08:44:50,786 INFO [download_results] running in process 1564846 2024-03-23 08:44:50,787 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 08:44:50,788 INFO [download_results] publishing log messages to tcp://*:5565 2024-03-23 08:44:50,789 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 08:44:50,790 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/21jul19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 08:44:54,992 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 08:45:19,396 INFO [download_results] hindcast 2019-07-21 results files from optimum-hindcast downloaded 2024-03-23 08:45:19,402 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 08:45:19,408 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-21'], host='localhost') 2024-03-23 08:45:20,283 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 09:03:09,179 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240323/12/ 2024-03-23 09:03:09,180 INFO [collect_weather] 2024-03-23 2.5km weather forecast 12 collection complete 2024-03-23 09:03:09,182 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-03-23 09:03:09,384 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-03-22'], host='localhost') 2024-03-23 09:03:09,389 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-03-22'], host='localhost') 2024-03-23 09:03:09,394 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-03-22'], host='localhost') 2024-03-23 09:03:09,398 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-03-22'], host='localhost') 2024-03-23 09:03:09,402 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2024-03-23 09:03:09,405 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2024-03-23 09:03:09,409 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2024-03-23 09:03:09,412 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2024-03-23 09:03:09,421 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2024-03-23 09:03:10,650 INFO [collect_NeahBay_ssh] running in process 1564934 2024-03-23 09:03:10,651 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:10,651 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563 2024-03-23 09:03:10,652 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:10,653 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-03-23 2024-03-23 09:03:10,668 INFO [collect_weather] running in process 1564936 2024-03-23 09:03:10,669 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:10,669 INFO [collect_weather] publishing log messages to tcp://*:5564 2024-03-23 09:03:10,670 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:10,684 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2024-03-23 09:03:10,751 INFO [download_live_ocean] running in process 1564935 2024-03-23 09:03:10,752 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:10,753 INFO [download_live_ocean] publishing log messages to tcp://*:5565 2024-03-23 09:03:10,754 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:10,755 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-03-23 2024-03-23 09:03:11,222 INFO [make_turbidity_file] running in process 1564933 2024-03-23 09:03:11,223 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:11,224 INFO [make_turbidity_file] publishing log messages to tcp://*:5566 2024-03-23 09:03:11,225 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:11,226 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-03-23 2024-03-23 09:03:11,296 INFO [collect_river_data] running in process 1564930 2024-03-23 09:03:11,297 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:11,298 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-03-23 09:03:11,298 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:11,299 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-03-22 2024-03-23 09:03:11,315 INFO [collect_river_data] running in process 1564932 2024-03-23 09:03:11,316 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:11,317 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-03-23 09:03:11,319 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:11,319 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-03-22 2024-03-23 09:03:11,321 INFO [collect_river_data] running in process 1564928 2024-03-23 09:03:11,322 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:11,323 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-03-23 09:03:11,323 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:11,324 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-03-22 2024-03-23 09:03:11,352 INFO [download_live_ocean] downloaded 10157619 bytes from https://liveocean.apl.uw.edu/output/f2024.03.23/ubc.nc to /results/forcing/LiveOcean/downloaded/20240323/low_passed_UBC.nc 2024-03-23 09:03:11,353 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2024-03-23 09:03:11,415 INFO [collect_river_data] running in process 1564931 2024-03-23 09:03:11,416 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:11,417 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-03-23 09:03:11,418 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:11,419 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-03-22 2024-03-23 09:03:11,612 INFO [crop_gribs] running in process 1564937 2024-03-23 09:03:11,613 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:11,614 INFO [crop_gribs] publishing log messages to tcp://*:5571 2024-03-23 09:03:11,615 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:11,616 INFO [crop_gribs] cropping 2024-03-23 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2024-03-23 09:03:11,626 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240323/18/ 2024-03-23 09:03:11,703 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2024-03-22&endDT=2024-03-22 "HTTP/1.1 200 OK" 2024-03-23 09:03:11,707 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-03-22 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2024-03-23 09:03:11,708 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-03-22 completed 2024-03-23 09:03:11,710 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-23 09:03:11,859 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2024-03-22&endDT=2024-03-22 "HTTP/1.1 200 OK" 2024-03-23 09:03:11,863 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-03-22 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2024-03-23 09:03:11,864 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-03-22 completed 2024-03-23 09:03:11,866 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-23 09:03:11,868 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2024-03-22&endDT=2024-03-22 "HTTP/1.1 200 OK" 2024-03-23 09:03:11,872 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-03-22 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2024-03-23 09:03:11,872 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-03-22 completed 2024-03-23 09:03:11,877 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-23 09:03:11,960 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2024-03-22&endDT=2024-03-22 "HTTP/1.1 200 OK" 2024-03-23 09:03:11,964 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-03-22 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2024-03-23 09:03:11,964 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-03-22 completed 2024-03-23 09:03:11,967 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-23 09:03:11,986 INFO [collect_NeahBay_ssh] 2024-03-23T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2024-03-23 09:03:11,988 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-03-23 09:03:11,998 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 09:03:12,402 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240323/low_passed_UBC.nc 2024-03-23 09:03:12,403 INFO [download_live_ocean] 2024-03-23 Live Ocean file for Salish Sea western boundary downloaded 2024-03-23 09:03:12,405 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2024-03-23 09:03:12,591 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-03-23'], host='localhost') 2024-03-23 09:03:14,068 INFO [make_turbidity_file] 2024-03-23 Fraser River turbidity file creation complete 2024-03-23 09:03:14,069 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2024-03-23 09:03:14,223 INFO [make_live_ocean_files] running in process 1565101 2024-03-23 09:03:14,224 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:14,225 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563 2024-03-23 09:03:14,226 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:14,227 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-03-23 Live Ocean run 2024-03-23 09:03:14,276 INFO [make_ssh_files] running in process 1565088 2024-03-23 09:03:14,278 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:14,279 INFO [make_ssh_files] publishing log messages to tcp://*:5565 2024-03-23 09:03:14,280 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:14,281 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-03-23 NOAA Neah Bay 06Z observation and forecast values 2024-03-23 09:03:18,714 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d19.nc 2024-03-23 09:03:18,888 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d20.nc 2024-03-23 09:03:19,066 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d21.nc 2024-03-23 09:03:19,244 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d22.nc 2024-03-23 09:03:19,451 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m03d23.nc 2024-03-23 09:03:19,625 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m03d24.nc 2024-03-23 09:03:19,799 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m03d25.nc 2024-03-23 09:03:19,979 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m03d26.nc 2024-03-23 09:03:20,089 INFO [make_ssh_files] sea surface height boundary file for 2024-03-23 nowcast run created 2024-03-23 09:03:20,092 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-03-23 09:03:20,102 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-03-23 09:03:20,104 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-03-23 09:03:21,089 INFO [make_runoff_file] running in process 1565146 2024-03-23 09:03:21,090 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:21,090 INFO [make_runoff_file] publishing log messages to tcp://*:5565 2024-03-23 09:03:21,091 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:21,117 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-03-23 09:03:21,209 INFO [make_runoff_file] 2024-03-23 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-03-23 09:03:21,211 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-03-23 09:03:21,788 INFO [make_v202111_runoff_file] running in process 1565144 2024-03-23 09:03:21,789 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:03:21,790 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5565 2024-03-23 09:03:21,791 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:03:21,791 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-03-22 2024-03-23 09:03:24,100 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-03-22: /results/forcing/rivers/R202108Dailies_y2024m03d22.nc 2024-03-23 09:03:24,100 INFO [make_v202111_runoff_file] 2024-03-22 runoff file creation completed 2024-03-23 09:03:24,103 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-03-23 09:03:54,812 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: / 2024-03-23 09:03:54,812 INFO [make_live_ocean_files] 2024-03-23 Live Ocean western boundary conditions files created 2024-03-23 09:03:54,814 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2024-03-23 09:09:51,711 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240323/12/ 2024-03-23 09:09:51,712 INFO [crop_gribs] 2024-03-23 12 GRIBs cropping complete 2024-03-23 09:09:51,715 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-03-23 09:09:51,916 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2024-03-23 09:09:53,820 INFO [grib_to_netcdf] running in process 1565196 2024-03-23 09:09:53,821 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:09:53,822 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563 2024-03-23 09:09:53,823 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:09:53,824 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-03-23 nowcast and forecast runs 2024-03-23 09:10:05,584 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m03d23.nc 2024-03-23 09:10:16,355 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m03d24.nc 2024-03-23 09:10:22,438 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m03d25.nc 2024-03-23 09:10:22,452 INFO [grib_to_netcdf] 2024-03-23 NEMO-atmos forcing file for nowcast+ created 2024-03-23 09:10:22,456 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-03-23 09:10:22,467 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2024-03-23 09:10:22,473 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2024-03-23 09:10:22,477 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2024-03-23 09:10:22,481 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2024-03-23 09:10:22,486 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2024-03-23 09:10:23,325 INFO [ping_erddap] running in process 1565238 2024-03-23 09:10:23,327 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:10:23,328 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-03-23 09:10:23,329 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:10:23,330 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2024-03-23 09:10:23,332 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-03-23 09:10:23,454 INFO [upload_forcing] running in process 1565237 2024-03-23 09:10:23,455 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:10:23,456 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-03-23 09:10:23,457 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:10:23,472 INFO [upload_forcing] running in process 1565236 2024-03-23 09:10:23,473 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:10:23,474 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-03-23 09:10:23,475 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:10:23,481 INFO [upload_forcing] running in process 1565233 2024-03-23 09:10:23,482 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:10:23,484 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-03-23 09:10:23,485 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:10:23,530 INFO [upload_forcing] running in process 1565235 2024-03-23 09:10:23,532 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:10:23,533 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-03-23 09:10:23,534 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:10:26,057 INFO [upload_forcing] nowcast+ 2024-03-23 forcing files upload to optimum-hindcast completed 2024-03-23 09:10:26,061 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-23 09:10:26,142 INFO [upload_forcing] nowcast+ 2024-03-23 forcing files upload to orcinus-nowcast-agrif completed 2024-03-23 09:10:26,147 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-23 09:10:33,590 INFO [upload_forcing] nowcast+ 2024-03-23 forcing files upload to robot.graham completed 2024-03-23 09:10:33,595 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-23 09:10:36,455 INFO [upload_forcing] nowcast+ 2024-03-23 forcing files upload to arbutus.cloud-nowcast completed 2024-03-23 09:10:36,460 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-23 09:10:36,472 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 09:10:37,407 INFO [make_forcing_links] running in process 1565273 2024-03-23 09:10:37,410 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:10:37,411 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-03-23 09:10:37,412 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:10:38,557 INFO [make_forcing_links] nowcast+ 2024-03-23 forcing file links on arbutus.cloud-nowcast created 2024-03-23 09:10:38,561 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-03-23 09:10:38,572 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-03-23'], host='arbutus.cloud-nowcast') 2024-03-23 09:10:40,653 INFO [run_NEMO] running in process 9729 2024-03-23 09:10:40,654 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:10:40,655 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-03-23 09:10:40,725 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:10:41,533 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/23mar24nowcast_2024-03-23T091040.710079-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-03-23 09:10:41,553 INFO [run_NEMO] nowcast NEMO run for 2024-03-23 on arbutus.cloud-nowcast started 2024-03-23 09:10:41,558 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-03-23 09:10:41,570 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-03-23 09:10:43,518 INFO [watch_NEMO] running in process 9981 2024-03-23 09:10:43,519 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:10:43,521 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-03-23 09:10:43,521 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:10:43,589 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-03-23 09:15:43,590 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17539624 = 2024-03-23 04:42:00 UTC, 19.6% complete 2024-03-23 09:20:43,591 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17540101 = 2024-03-23 10:00:00 UTC, 41.7% complete 2024-03-23 09:25:43,592 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17540564 = 2024-03-23 15:08:40 UTC, 63.1% complete 2024-03-23 09:30:43,596 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17541071 = 2024-03-23 20:46:40 UTC, 86.6% complete 2024-03-23 09:35:43,603 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2024-03-23 09:35:43,607 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-03-23 09:35:43,814 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 09:35:43,820 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 09:35:44,720 INFO [make_forcing_links] running in process 1565966 2024-03-23 09:35:44,721 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:35:44,722 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-03-23 09:35:44,724 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:35:44,726 INFO [download_results] running in process 1565967 2024-03-23 09:35:44,727 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:35:44,728 INFO [download_results] publishing log messages to tcp://*:5565 2024-03-23 09:35:44,729 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:35:44,730 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/23mar24 to /results/SalishSea/nowcast-blue.202111 2024-03-23 09:35:45,703 INFO [make_forcing_links] ssh 2024-03-23 forcing file links on arbutus.cloud-nowcast created 2024-03-23 09:35:45,708 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-03-23 09:35:45,721 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-03-23'], host='arbutus.cloud-nowcast') 2024-03-23 09:35:48,869 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/23mar24forecast_2024-03-23T093547.997913-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-03-23 09:35:48,890 INFO [run_NEMO] forecast NEMO run for 2024-03-23 on arbutus.cloud-nowcast started 2024-03-23 09:35:48,894 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-03-23 09:35:48,906 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-03-23 09:35:51,642 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-03-23 09:39:56,002 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 09:39:56,211 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-26'], host='localhost') 2024-03-23 09:39:56,218 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 09:39:56,224 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 09:39:57,219 INFO [download_results] running in process 1565997 2024-03-23 09:39:57,220 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:39:57,222 INFO [download_results] publishing log messages to tcp://*:5566 2024-03-23 09:39:57,223 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:39:57,224 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/26jul19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 09:40:01,299 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 09:40:29,744 INFO [download_results] hindcast 2019-07-26 results files from optimum-hindcast downloaded 2024-03-23 09:40:29,752 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 09:40:29,763 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-26'], host='localhost') 2024-03-23 09:40:30,693 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 09:40:30,706 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-jul', 'robot.graham'], host='localhost') 2024-03-23 09:40:31,600 INFO [archive_tarball] running in process 1566045 2024-03-23 09:40:31,604 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:40:31,605 INFO [archive_tarball] publishing log messages to tcp://*:5563 2024-03-23 09:40:31,606 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:40:31,607 INFO [archive_tarball] creating /ocean/dlatorne/rivers-jul19.tar from /ocean/sallen/timbucktwo/rivers/*jul19/ 2024-03-23 09:40:51,596 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17541785 = 2024-03-24 04:42:40 UTC, 13.1% complete 2024-03-23 09:43:49,240 INFO [archive_tarball] creating /ocean/dlatorne/rivers-jul19.index from /ocean/dlatorne/rivers-jul19.tar 2024-03-23 09:43:49,302 INFO [archive_tarball] rsync-ing /ocean/dlatorne/rivers-jul19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/rivers/ 2024-03-23 09:45:52,238 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17542260 = 2024-03-24 09:59:20 UTC, 27.8% complete 2024-03-23 09:50:51,888 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17542714 = 2024-03-24 15:02:00 UTC, 41.8% complete 2024-03-23 09:51:34,549 INFO [archive_tarball] hindcast *jul19 results files archived to robot.graham 2024-03-23 09:51:34,554 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-03-23 09:55:46,319 INFO [download_results] nowcast 2024-03-23 results files from arbutus.cloud-nowcast downloaded 2024-03-23 09:55:46,324 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 09:55:46,341 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 09:55:46,347 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-03-22'], host='localhost') 2024-03-23 09:55:46,351 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 09:55:48,122 INFO [make_CHS_currents_file] running in process 1566403 2024-03-23 09:55:48,124 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:55:48,124 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-03-23 09:55:48,125 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:55:49,348 INFO [make_plots] running in process 1566400 2024-03-23 09:55:49,350 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:55:49,351 INFO [make_plots] publishing log messages to tcp://*:5565 2024-03-23 09:55:49,352 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:55:49,360 INFO [make_plots] running in process 1566402 2024-03-23 09:55:49,386 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 09:55:49,399 INFO [make_plots] publishing log messages to tcp://*:5566 2024-03-23 09:55:49,401 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-03-23 09:55:50,039 INFO [make_plots] preparing render list for 2024-03-23 NEMO nowcast-blue research figures 2024-03-23 09:55:50,041 INFO [make_plots] preparing render list for 2024-03-22 NEMO nowcast-blue comparison figures 2024-03-23 09:55:50,091 CRITICAL [make_plots] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 207, in make_plots fig_functions = _prep_comparison_fig_functions( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 690, in _prep_comparison_fig_functions dev_grid_T_hr = _results_dataset("1h", "grid_T", dev_results_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 344, in _results_dataset return nc.Dataset(filepaths[0]) ~~~~~~~~~^^^ IndexError: list index out of range 2024-03-23 09:55:51,039 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-blue research render list 2024-03-23 09:55:51,601 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17543174 = 2024-03-24 20:08:40 UTC, 56.0% complete 2024-03-23 09:55:51,660 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-blue research render list 2024-03-23 09:55:51,661 INFO [make_plots] added Currents_sections_and_surface figure to 2024-03-23 NEMO nowcast-blue research render list 2024-03-23 09:55:51,662 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-03-23 NEMO nowcast-blue research render list 2024-03-23 09:55:51,663 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-03-23 NEMO nowcast-blue research render list 2024-03-23 09:55:51,664 INFO [make_plots] starting to render nemo nowcast research 23mar24 figures 2024-03-23 09:56:17,399 INFO [make_CHS_currents_file] Made CHS currents file for 2024-03-23 for nowcast 2024-03-23 09:56:17,402 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-03-23 09:57:05,496 INFO [make_plots] finished rendering nemo nowcast research 23mar24 figures 2024-03-23 09:57:05,499 INFO [make_plots] nemo research plots for 2024-03-23 nowcast completed 2024-03-23 09:57:05,516 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-03-23 10:00:51,603 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17543637 = 2024-03-25 01:17:20 UTC, 70.3% complete 2024-03-23 10:05:51,601 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17544119 = 2024-03-25 06:38:40 UTC, 85.1% complete 2024-03-23 10:10:51,601 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17544588 = 2024-03-25 11:51:20 UTC, 99.6% complete 2024-03-23 10:15:51,604 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2024-03-23 10:15:51,608 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-03-23 10:15:51,815 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-03-23'], host='arbutus.cloud-nowcast') 2024-03-23 10:15:51,822 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-03-23'], host='arbutus.cloud-nowcast') 2024-03-23 10:15:51,826 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:15:51,831 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:15:51,837 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:15:51,841 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:15:51,845 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:15:52,868 INFO [download_results] running in process 1566548 2024-03-23 10:15:52,870 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:15:52,871 INFO [download_results] publishing log messages to tcp://*:5563 2024-03-23 10:15:52,872 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:15:52,873 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/23mar24 to /results/SalishSea/forecast.202111 2024-03-23 10:15:52,875 INFO [upload_forcing] running in process 1566546 2024-03-23 10:15:52,876 INFO [upload_forcing] running in process 1566547 2024-03-23 10:15:52,877 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:15:52,878 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:15:52,880 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-03-23 10:15:52,881 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-03-23 10:15:52,882 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:15:52,883 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:15:52,884 INFO [upload_forcing] running in process 1566543 2024-03-23 10:15:52,885 INFO [upload_forcing] running in process 1566544 2024-03-23 10:15:52,886 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:15:52,887 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:15:52,887 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-03-23 10:15:52,888 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-03-23 10:15:52,889 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:15:52,890 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:15:53,530 INFO [upload_forcing] turbidity 2024-03-23 forcing files upload to optimum-hindcast completed 2024-03-23 10:15:53,534 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-23 10:15:53,858 INFO [upload_forcing] turbidity 2024-03-23 forcing files upload to orcinus-nowcast-agrif completed 2024-03-23 10:15:53,863 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-23 10:15:53,884 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:15:53,903 INFO [upload_forcing] turbidity 2024-03-23 forcing files upload to arbutus.cloud-nowcast completed 2024-03-23 10:15:53,907 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-23 10:15:53,925 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:15:54,828 INFO [make_forcing_links] running in process 1566580 2024-03-23 10:15:54,830 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:15:54,831 INFO [make_forcing_links] publishing log messages to tcp://*:5566 2024-03-23 10:15:54,832 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:15:54,906 INFO [make_forcing_links] running in process 1566582 2024-03-23 10:15:54,908 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:15:54,909 INFO [make_forcing_links] publishing log messages to tcp://*:5567 2024-03-23 10:15:54,910 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:15:55,803 INFO [make_forcing_links] nowcast-agrif 2024-03-23 forcing file links on orcinus-nowcast-agrif created 2024-03-23 10:15:55,808 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-03-23 10:15:55,831 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:15:56,082 INFO [make_forcing_links] nowcast-green 2024-03-23 forcing file links on arbutus.cloud-nowcast created 2024-03-23 10:15:56,085 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-03-23 10:15:56,103 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-03-23'], host='arbutus.cloud-nowcast') 2024-03-23 10:15:56,830 INFO [run_NEMO_agrif] running in process 1566596 2024-03-23 10:15:56,833 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:15:56,834 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5566 2024-03-23 10:15:56,836 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:15:57,963 INFO [upload_forcing] turbidity 2024-03-23 forcing files upload to robot.graham completed 2024-03-23 10:15:57,967 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-23 10:15:58,623 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-03-23 10:15:59,664 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/23mar24nowcast-green_2024-03-23T101558.696697-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-03-23 10:15:59,685 INFO [run_NEMO] nowcast-green NEMO run for 2024-03-23 on arbutus.cloud-nowcast started 2024-03-23 10:15:59,911 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-03-23 10:15:59,932 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2024-03-23 10:16:02,593 CRITICAL [run_NEMO_agrif] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 110, in run_NEMO_agrif prev_run_namelists_info = _get_prev_run_namelists_info( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 156, in _get_prev_run_namelists_info sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/sftp_client.py", line 811, in get size = self.getfo(remotepath, fl, callback, prefetch) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/sftp_client.py", line 782, in getfo file_size = self.stat(remotepath).st_size ^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/sftp_client.py", line 493, in stat t, msg = self._request(CMD_STAT, path) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/sftp_client.py", line 822, in _request return self._read_response(num) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/sftp_client.py", line 874, in _read_response self._convert_status(msg) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/sftp_client.py", line 903, in _convert_status raise IOError(errno.ENOENT, text) FileNotFoundError: [Errno 2] No such file 2024-03-23 10:16:02,970 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-03-23 10:16:38,001 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-03-23 10:16:38,024 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-03-23'], host='arbutus.cloud-nowcast') 2024-03-23 10:16:49,232 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-03-23 10:16:49,256 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-03-23 10:17:01,116 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-03-23 10:21:03,457 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17539353 = 2024-03-23 01:41:20 UTC, 7.0% complete 2024-03-23 10:22:01,113 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete 2024-03-23 10:26:03,340 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17539525 = 2024-03-23 03:36:00 UTC, 15.0% complete 2024-03-23 10:27:01,097 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 183, 42.4% complete 2024-03-23 10:30:02,096 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 10:30:02,310 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-01'], host='localhost') 2024-03-23 10:30:02,316 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 10:30:02,321 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 10:30:03,307 INFO [download_results] running in process 1566675 2024-03-23 10:30:03,309 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:30:03,309 INFO [download_results] publishing log messages to tcp://*:5566 2024-03-23 10:30:03,311 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:30:03,312 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/01aug19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 10:30:07,417 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 10:30:32,286 INFO [download_results] hindcast 2019-08-01 results files from optimum-hindcast downloaded 2024-03-23 10:30:32,292 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 10:30:32,310 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-01'], host='localhost') 2024-03-23 10:30:33,214 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 10:31:03,547 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17539697 = 2024-03-23 05:30:40 UTC, 23.0% complete 2024-03-23 10:32:01,107 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 279, 64.6% complete 2024-03-23 10:36:03,562 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17539869 = 2024-03-23 07:25:20 UTC, 30.9% complete 2024-03-23 10:37:01,103 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 372, 86.1% complete 2024-03-23 10:41:03,540 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17540042 = 2024-03-23 09:20:40 UTC, 39.0% complete 2024-03-23 10:42:01,056 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-03-23 10:42:01,059 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-03-23 10:42:01,312 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:42:01,318 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-03-23'], host='arbutus.cloud-nowcast') 2024-03-23 10:42:02,249 INFO [download_wwatch3_results] running in process 1566747 2024-03-23 10:42:02,251 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:42:02,252 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565 2024-03-23 10:42:02,253 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:42:12,424 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-03-23 10:42:12,443 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-03-23 10:42:24,197 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-03-23 10:46:04,190 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17540214 = 2024-03-23 11:15:20 UTC, 46.9% complete 2024-03-23 10:47:13,613 INFO [download_wwatch3_results] nowcast 2024-03-23 results files from arbutus.cloud-nowcast downloaded 2024-03-23 10:47:13,617 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-03-23 10:47:18,792 INFO [download_results] forecast 2024-03-23 results files from arbutus.cloud-nowcast downloaded 2024-03-23 10:47:18,797 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 10:47:18,819 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:47:20,748 INFO [make_CHS_currents_file] running in process 1566767 2024-03-23 10:47:20,749 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:47:20,750 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-03-23 10:47:20,751 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:47:23,564 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete 2024-03-23 10:48:01,848 INFO [make_CHS_currents_file] Made CHS currents file for 2024-03-23 for forecast 2024-03-23 10:48:01,851 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-03-23 10:48:01,872 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:48:02,653 INFO [update_forecast_datasets] running in process 1566792 2024-03-23 10:48:02,655 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:48:02,656 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-03-23 10:48:02,657 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:48:02,658 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/23mar24 run 2024-03-23 10:48:02,920 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/23mar24 run: /results/SalishSea/rolling-forecasts/nemo 2024-03-23 10:48:02,921 INFO [update_forecast_datasets] nemo 2024-03-23 forecast rolling forecast datasets updated 2024-03-23 10:48:02,922 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-03-23 10:48:02,940 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-03-23 10:48:02,945 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:48:02,949 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 10:48:03,822 INFO [ping_erddap] running in process 1566797 2024-03-23 10:48:03,823 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:48:03,824 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-03-23 10:48:03,825 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:48:03,835 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-03-23 10:48:03,836 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-03-23 10:48:04,889 INFO [make_surface_current_tiles] running in process 1566800 2024-03-23 10:48:04,890 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:48:04,891 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2024-03-23 10:48:04,892 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:48:04,926 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-03-23 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/23mar24 2024-03-23 10:48:06,277 INFO [make_plots] running in process 1566799 2024-03-23 10:48:06,279 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 10:48:06,281 INFO [make_plots] publishing log messages to tcp://*:5565 2024-03-23 10:48:06,283 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-03-23 10:48:06,921 INFO [make_plots] preparing render list for 2024-03-23 NEMO forecast publish figures 2024-03-23 10:48:07,035 INFO [make_plots] added Website_thumbnail figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,036 INFO [make_plots] added Threshold_website figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,037 INFO [make_plots] added PA_tidal_predictions figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,038 INFO [make_plots] added Boundary Bay figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,039 INFO [make_plots] added Campbell River figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,040 INFO [make_plots] added Cherry Point figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,043 INFO [make_plots] added Friday Harbor figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,043 INFO [make_plots] added Halfmoon Bay figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,044 INFO [make_plots] added Nanaimo figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,045 INFO [make_plots] added Neah Bay figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,046 INFO [make_plots] added New Westminster figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,047 INFO [make_plots] added Patricia Bay figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,048 INFO [make_plots] added Point Atkinson figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,050 INFO [make_plots] added Port Renfrew figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,051 INFO [make_plots] added Sand Heads figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,052 INFO [make_plots] added Sandy Cove figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,053 INFO [make_plots] added Squamish figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,055 INFO [make_plots] added Victoria figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,056 INFO [make_plots] added Woodwards Landing figure to 2024-03-23 NEMO forecast publish render list 2024-03-23 10:48:07,056 INFO [make_plots] starting to render nemo forecast publish 23mar24 figures 2024-03-23 10:51:02,906 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17540385 = 2024-03-23 13:09:20 UTC, 54.8% complete 2024-03-23 10:52:23,566 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 183, 28.2% complete 2024-03-23 10:56:02,907 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17540558 = 2024-03-23 15:04:40 UTC, 62.9% complete 2024-03-23 10:57:23,566 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 273, 42.1% complete 2024-03-23 11:01:02,910 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17540730 = 2024-03-23 16:59:20 UTC, 70.8% complete 2024-03-23 11:01:53,992 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/23mar24 2024-03-23 11:02:23,568 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 372, 57.4% complete 2024-03-23 11:05:44,539 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/23mar24 2024-03-23 11:05:44,549 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-03-23 forecast into /results/nowcast-sys/figures/surface_currents/forecast/23mar24 2024-03-23 11:05:44,551 INFO [make_surface_current_tiles] surface current tile figures for 2024-03-23 forecast completed 2024-03-23 11:05:44,874 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-03-23 11:05:50,805 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-03-23 11:05:50,806 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-03-23 11:06:02,921 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17540902 = 2024-03-23 18:54:00 UTC, 78.8% complete 2024-03-23 11:06:26,045 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:07:23,569 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 468, 72.2% complete 2024-03-23 11:08:15,932 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:08:16,355 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:08:52,216 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:10:03,974 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:10:39,721 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:11:02,911 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17541075 = 2024-03-23 20:49:20 UTC, 86.8% complete 2024-03-23 11:11:16,798 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:11:53,962 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:12:23,570 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 567, 87.5% complete 2024-03-23 11:12:29,482 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:13:06,260 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:13:42,920 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:13:43,330 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:14:20,218 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:14:56,043 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-03-18 00:00:00Z to 2024-03-25 11:55:00Z 2024-03-23 11:15:32,928 INFO [make_plots] finished rendering nemo forecast publish 23mar24 figures 2024-03-23 11:15:32,930 INFO [make_plots] nemo publish plots for 2024-03-23 forecast completed 2024-03-23 11:15:32,938 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-03-23 11:15:33,115 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:15:35,681 INFO [make_feeds] running in process 1568920 2024-03-23 11:15:35,682 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:15:35,683 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-03-23 11:15:35,684 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:16:02,912 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17541247 = 2024-03-23 22:44:00 UTC, 94.8% complete 2024-03-23 11:16:04,281 INFO [make_feeds] ATOM feeds for 2024-03-23 forecast run completed 2024-03-23 11:16:04,285 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-03-23 11:17:23,571 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-03-23 11:20:08,204 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 11:20:08,571 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-06'], host='localhost') 2024-03-23 11:20:08,578 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 11:20:08,583 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 11:20:09,558 INFO [download_results] running in process 1568953 2024-03-23 11:20:09,560 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:20:09,561 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 11:20:09,563 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:20:09,564 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/06aug19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 11:20:13,557 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 11:20:38,323 INFO [download_results] hindcast 2019-08-06 results files from optimum-hindcast downloaded 2024-03-23 11:20:38,330 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 11:20:38,502 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-06'], host='localhost') 2024-03-23 11:20:39,343 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 11:21:03,216 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17541360 = 2024-03-23 23:59:20 UTC, 100.0% complete 2024-03-23 11:22:23,573 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-03-23 11:22:23,576 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-03-23 11:22:23,949 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:22:24,873 INFO [download_wwatch3_results] running in process 1569001 2024-03-23 11:22:24,875 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:22:24,876 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562 2024-03-23 11:22:24,877 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:24:58,948 INFO [download_wwatch3_results] forecast 2024-03-23 results files from arbutus.cloud-nowcast downloaded 2024-03-23 11:24:58,951 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-03-23 11:24:59,130 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:25:00,065 INFO [update_forecast_datasets] running in process 1569010 2024-03-23 11:25:00,066 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:25:00,068 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2024-03-23 11:25:00,069 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:25:00,070 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/23mar24 run 2024-03-23 11:25:00,072 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/23mar24 run 2024-03-23 11:25:00,129 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/23mar24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-03-23 11:25:00,130 INFO [update_forecast_datasets] wwatch3 2024-03-23 forecast rolling forecast datasets updated 2024-03-23 11:25:00,133 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-03-23 11:25:00,315 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-03-23 11:25:01,164 INFO [ping_erddap] running in process 1569015 2024-03-23 11:25:01,166 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:25:01,167 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-03-23 11:25:01,168 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:25:01,169 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-03-23 11:25:01,172 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-03-23 11:25:01,352 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:25:04,740 INFO [make_plots] running in process 1569020 2024-03-23 11:25:04,741 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:25:04,742 INFO [make_plots] publishing log messages to tcp://*:5562 2024-03-23 11:25:04,743 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:25:04,744 INFO [make_plots] preparing render list for 2024-03-23 WaveWatch3 forecast publish figures 2024-03-23 11:25:04,745 INFO [make_plots] added Halibut Bank figure to 2024-03-23 WaveWatch3 forecast publish render list 2024-03-23 11:25:04,746 INFO [make_plots] added Sentry Shoal figure to 2024-03-23 WaveWatch3 forecast publish render list 2024-03-23 11:25:04,747 INFO [make_plots] starting to render wwatch3 forecast publish 23mar24 figures 2024-03-23 11:25:04,955 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-03-23 11:25:13,173 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-03-23 11:25:19,984 INFO [make_plots] finished rendering wwatch3 forecast publish 23mar24 figures 2024-03-23 11:25:19,985 INFO [make_plots] wwatch3 publish plots for 2024-03-23 forecast completed 2024-03-23 11:25:19,989 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-03-23 11:26:02,974 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2024-03-23 11:26:02,977 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-03-23 11:26:03,345 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:26:04,324 INFO [download_results] running in process 1569058 2024-03-23 11:26:04,326 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:26:04,327 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 11:26:04,328 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:26:04,329 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/23mar24 to /results2/SalishSea/nowcast-green.202111 2024-03-23 11:59:41,021 INFO [download_results] nowcast-green 2024-03-23 results files from arbutus.cloud-nowcast downloaded 2024-03-23 11:59:41,028 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 11:59:41,211 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:59:41,217 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2024-03-23 11:59:41,221 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:59:41,225 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:59:41,229 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-03-23'], host='localhost') 2024-03-23 11:59:42,166 INFO [ping_erddap] running in process 1569250 2024-03-23 11:59:42,167 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:59:42,170 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-03-23 11:59:42,171 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:59:42,180 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2024-03-23 11:59:42,182 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-03-23 11:59:43,494 INFO [make_averaged_dataset] running in process 1569252 2024-03-23 11:59:43,496 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:59:43,497 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562 2024-03-23 11:59:43,497 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:59:43,498 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Mar-2024 biology 2024-03-23 11:59:43,499 INFO [make_averaged_dataset] running in process 1569253 2024-03-23 11:59:43,501 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:59:43,502 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565 2024-03-23 11:59:43,503 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:59:43,504 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Mar-2024 chemistry 2024-03-23 11:59:43,505 INFO [make_averaged_dataset] running in process 1569251 2024-03-23 11:59:43,506 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:59:43,507 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566 2024-03-23 11:59:43,507 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:59:43,508 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Mar-2024 physics 2024-03-23 11:59:45,126 INFO [make_plots] running in process 1569248 2024-03-23 11:59:45,127 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 11:59:45,128 INFO [make_plots] publishing log messages to tcp://*:5567 2024-03-23 11:59:45,129 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-03-23 11:59:45,690 INFO [make_plots] preparing render list for 2024-03-23 NEMO nowcast-green research figures 2024-03-23 11:59:47,401 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:48,121 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:48,825 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:49,535 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:50,311 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:51,022 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:51,745 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:52,560 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:53,370 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:54,164 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:54,967 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:55,675 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:55,945 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:55,946 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:55,947 INFO [make_plots] added mesodinium_flagellates_timeseries figure to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:55,947 INFO [make_plots] added mesozoo_microzoo_timeseries figure to 2024-03-23 NEMO nowcast-green research render list 2024-03-23 11:59:55,948 INFO [make_plots] starting to render nemo nowcast-green research 23mar24 figures 2024-03-23 12:01:13,777 INFO [make_averaged_dataset] day-averaged dataset for 23-Mar-2024 physics created 2024-03-23 12:01:13,780 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-03-23 12:02:12,031 INFO [make_averaged_dataset] day-averaged dataset for 23-Mar-2024 chemistry created 2024-03-23 12:02:12,035 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-03-23 12:02:39,796 INFO [make_averaged_dataset] day-averaged dataset for 23-Mar-2024 biology created 2024-03-23 12:02:39,799 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-03-23 12:10:14,793 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 12:10:15,184 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-11'], host='localhost') 2024-03-23 12:10:15,192 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 12:10:15,196 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 12:10:16,436 INFO [download_results] running in process 1569419 2024-03-23 12:10:16,449 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 12:10:16,467 INFO [download_results] publishing log messages to tcp://*:5563 2024-03-23 12:10:16,485 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 12:10:16,487 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/11aug19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 12:10:20,444 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 12:10:51,288 INFO [download_results] hindcast 2019-08-11 results files from optimum-hindcast downloaded 2024-03-23 12:10:51,295 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 12:10:51,479 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-11'], host='localhost') 2024-03-23 12:10:52,383 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 12:55:20,968 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 12:55:21,365 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-16'], host='localhost') 2024-03-23 12:55:21,371 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 12:55:21,376 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 12:55:22,489 INFO [download_results] running in process 1569805 2024-03-23 12:55:22,491 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 12:55:22,492 INFO [download_results] publishing log messages to tcp://*:5565 2024-03-23 12:55:22,494 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 12:55:22,495 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/16aug19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 12:55:26,546 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 12:55:54,629 INFO [download_results] hindcast 2019-08-16 results files from optimum-hindcast downloaded 2024-03-23 12:55:54,656 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 12:55:54,840 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-16'], host='localhost') 2024-03-23 12:55:55,802 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 13:07:24,427 INFO [make_plots] finished rendering nemo nowcast-green research 23mar24 figures 2024-03-23 13:07:24,596 INFO [make_plots] nemo research plots for 2024-03-23 nowcast-green completed 2024-03-23 13:07:24,667 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-03-23 13:45:27,183 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 13:45:27,588 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-21'], host='localhost') 2024-03-23 13:45:27,620 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 13:45:27,624 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 13:45:28,660 INFO [download_results] running in process 1570106 2024-03-23 13:45:28,661 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 13:45:28,661 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 13:45:28,662 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 13:45:28,663 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/21aug19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 13:45:32,784 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 13:45:57,536 INFO [download_results] hindcast 2019-08-21 results files from optimum-hindcast downloaded 2024-03-23 13:45:57,544 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 13:45:57,770 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-21'], host='localhost') 2024-03-23 13:45:59,226 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 14:45:34,248 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 14:45:34,670 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-26'], host='localhost') 2024-03-23 14:45:34,677 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 14:45:34,681 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 14:45:35,641 INFO [download_results] running in process 1570342 2024-03-23 14:45:35,642 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 14:45:35,643 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 14:45:35,644 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 14:45:35,645 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/26aug19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 14:45:39,833 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 14:46:07,530 INFO [download_results] hindcast 2019-08-26 results files from optimum-hindcast downloaded 2024-03-23 14:46:07,539 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 14:46:07,752 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-26'], host='localhost') 2024-03-23 14:46:08,574 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 14:46:08,783 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-aug', 'robot.graham'], host='localhost') 2024-03-23 14:46:09,596 INFO [archive_tarball] running in process 1570386 2024-03-23 14:46:09,597 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 14:46:09,599 INFO [archive_tarball] publishing log messages to tcp://*:5562 2024-03-23 14:46:09,600 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-03-23 14:46:09,601 INFO [archive_tarball] creating /ocean/dlatorne/rivers-aug19.tar from /ocean/sallen/timbucktwo/rivers/*aug19/ 2024-03-23 14:50:25,093 INFO [archive_tarball] creating /ocean/dlatorne/rivers-aug19.index from /ocean/dlatorne/rivers-aug19.tar 2024-03-23 14:50:25,151 INFO [archive_tarball] rsync-ing /ocean/dlatorne/rivers-aug19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/rivers/ 2024-03-23 15:03:44,307 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240323/18/ 2024-03-23 15:03:44,308 INFO [collect_weather] 2024-03-23 2.5km weather forecast 18 collection complete 2024-03-23 15:03:44,310 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-03-23 15:03:44,750 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2024-03-23 15:03:44,755 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2024-03-23 15:03:44,760 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2024-03-23 15:03:44,764 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-03-24'], host='localhost') 2024-03-23 15:03:45,829 INFO [download_weather] running in process 1570504 2024-03-23 15:03:45,830 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 15:03:45,831 INFO [download_weather] publishing log messages to tcp://*:5564 2024-03-23 15:03:45,832 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-03-23 15:03:45,833 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240323 2024-03-23 15:03:45,837 INFO [download_weather] running in process 1570506 2024-03-23 15:03:45,838 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 15:03:45,839 INFO [download_weather] publishing log messages to tcp://*:5565 2024-03-23 15:03:45,840 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-03-23 15:03:45,841 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240323 2024-03-23 15:03:45,842 INFO [collect_weather] running in process 1570507 2024-03-23 15:03:45,843 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 15:03:45,844 INFO [collect_weather] publishing log messages to tcp://*:5566 2024-03-23 15:03:45,845 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-03-23 15:03:45,867 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2024-03-23 15:03:47,259 INFO [crop_gribs] running in process 1570508 2024-03-23 15:03:47,260 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 15:03:47,260 INFO [crop_gribs] publishing log messages to tcp://*:5567 2024-03-23 15:03:47,261 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-03-23 15:03:47,262 INFO [crop_gribs] cropping 2024-03-24 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2024-03-23 15:03:47,274 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240324/00/ 2024-03-23 15:06:59,885 INFO [archive_tarball] hindcast *aug19 results files archived to robot.graham 2024-03-23 15:06:59,887 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-03-23 15:08:32,969 INFO [download_weather] 2024-03-23 1km weather forecast 00 downloads complete 2024-03-23 15:08:32,973 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-03-23 15:08:34,619 INFO [download_weather] 2024-03-23 1km weather forecast 12 downloads complete 2024-03-23 15:08:34,622 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-03-23 15:10:04,639 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240323/18/ 2024-03-23 15:10:04,640 INFO [crop_gribs] 2024-03-23 18 GRIBs cropping complete 2024-03-23 15:10:04,642 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-03-23 15:35:40,501 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 15:35:40,908 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-01'], host='localhost') 2024-03-23 15:35:40,915 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 15:35:40,920 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 15:35:41,888 INFO [download_results] running in process 1570651 2024-03-23 15:35:41,889 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 15:35:41,890 INFO [download_results] publishing log messages to tcp://*:5563 2024-03-23 15:35:41,891 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 15:35:41,892 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/01sep19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 15:35:45,921 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 15:36:10,480 INFO [download_results] hindcast 2019-09-01 results files from optimum-hindcast downloaded 2024-03-23 15:36:10,487 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 15:36:10,701 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-01'], host='localhost') 2024-03-23 15:36:11,616 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 16:20:46,375 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 16:20:46,766 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-06'], host='localhost') 2024-03-23 16:20:46,773 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 16:20:46,777 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 16:20:47,761 INFO [download_results] running in process 1571006 2024-03-23 16:20:47,763 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 16:20:47,764 INFO [download_results] publishing log messages to tcp://*:5564 2024-03-23 16:20:47,766 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 16:20:47,767 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/06sep19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 16:20:51,765 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 16:21:16,496 INFO [download_results] hindcast 2019-09-06 results files from optimum-hindcast downloaded 2024-03-23 16:21:16,503 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 16:21:16,713 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-06'], host='localhost') 2024-03-23 16:21:17,570 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 17:10:52,456 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 17:10:52,856 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-11'], host='localhost') 2024-03-23 17:10:52,863 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 17:10:52,867 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 17:10:53,784 INFO [download_results] running in process 1571518 2024-03-23 17:10:53,785 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 17:10:53,786 INFO [download_results] publishing log messages to tcp://*:5564 2024-03-23 17:10:53,787 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 17:10:53,788 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/11sep19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 17:10:58,060 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 17:11:22,737 INFO [download_results] hindcast 2019-09-11 results files from optimum-hindcast downloaded 2024-03-23 17:11:22,745 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 17:11:22,962 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-11'], host='localhost') 2024-03-23 17:11:23,798 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 18:00:58,674 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 18:00:59,076 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-16'], host='localhost') 2024-03-23 18:00:59,083 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 18:00:59,088 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 18:01:00,021 INFO [download_results] running in process 1571816 2024-03-23 18:01:00,022 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 18:01:00,022 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 18:01:00,023 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 18:01:00,024 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/16sep19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 18:01:04,219 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 18:01:28,663 INFO [download_results] hindcast 2019-09-16 results files from optimum-hindcast downloaded 2024-03-23 18:01:28,670 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 18:01:28,879 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-16'], host='localhost') 2024-03-23 18:01:29,695 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 18:51:04,869 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 18:51:05,265 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-21'], host='localhost') 2024-03-23 18:51:05,272 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 18:51:05,276 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 18:51:06,167 INFO [download_results] running in process 1572255 2024-03-23 18:51:06,168 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 18:51:06,169 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 18:51:06,170 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 18:51:06,170 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/21sep19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 18:51:10,159 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 18:51:34,891 INFO [download_results] hindcast 2019-09-21 results files from optimum-hindcast downloaded 2024-03-23 18:51:34,897 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 18:51:35,101 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-21'], host='localhost') 2024-03-23 18:51:35,902 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 19:41:11,017 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 19:41:11,419 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-26'], host='localhost') 2024-03-23 19:41:11,426 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 19:41:11,431 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 19:41:12,400 INFO [download_results] running in process 1572477 2024-03-23 19:41:12,402 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 19:41:12,403 INFO [download_results] publishing log messages to tcp://*:5564 2024-03-23 19:41:12,404 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 19:41:12,405 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/26sep19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 19:41:16,593 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 19:41:41,167 INFO [download_results] hindcast 2019-09-26 results files from optimum-hindcast downloaded 2024-03-23 19:41:41,175 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 19:41:41,390 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-26'], host='localhost') 2024-03-23 19:41:42,251 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 19:41:42,464 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-sep', 'robot.graham'], host='localhost') 2024-03-23 19:41:43,293 INFO [archive_tarball] running in process 1572522 2024-03-23 19:41:43,295 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 19:41:43,296 INFO [archive_tarball] publishing log messages to tcp://*:5562 2024-03-23 19:41:43,297 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-03-23 19:41:43,298 INFO [archive_tarball] creating /ocean/dlatorne/rivers-sep19.tar from /ocean/sallen/timbucktwo/rivers/*sep19/ 2024-03-23 19:44:29,136 INFO [archive_tarball] creating /ocean/dlatorne/rivers-sep19.index from /ocean/dlatorne/rivers-sep19.tar 2024-03-23 19:44:29,200 INFO [archive_tarball] rsync-ing /ocean/dlatorne/rivers-sep19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/rivers/ 2024-03-23 19:52:20,429 INFO [archive_tarball] hindcast *sep19 results files archived to robot.graham 2024-03-23 19:52:20,434 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-03-23 20:26:16,910 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 20:26:17,303 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-01'], host='localhost') 2024-03-23 20:26:17,310 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 20:26:17,315 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 20:26:18,227 INFO [download_results] running in process 1572954 2024-03-23 20:26:18,228 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 20:26:18,229 INFO [download_results] publishing log messages to tcp://*:5562 2024-03-23 20:26:18,230 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 20:26:18,231 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/01oct19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 20:26:22,368 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 20:26:46,910 INFO [download_results] hindcast 2019-10-01 results files from optimum-hindcast downloaded 2024-03-23 20:26:46,917 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 20:26:47,125 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-01'], host='localhost') 2024-03-23 20:26:47,920 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 21:04:54,563 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240324/00/ 2024-03-23 21:04:54,564 INFO [collect_weather] 2024-03-24 2.5km weather forecast 00 collection complete 2024-03-23 21:04:54,566 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-03-23 21:04:54,970 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2024-03-23 21:04:54,976 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-03-24'], host='localhost') 2024-03-23 21:04:55,979 INFO [collect_weather] running in process 1573424 2024-03-23 21:04:55,980 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 21:04:55,981 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-03-23 21:04:55,981 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-03-23 21:04:55,996 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2024-03-23 21:04:56,915 INFO [crop_gribs] running in process 1573426 2024-03-23 21:04:56,916 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 21:04:56,918 INFO [crop_gribs] publishing log messages to tcp://*:5564 2024-03-23 21:04:56,919 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-03-23 21:04:56,919 INFO [crop_gribs] cropping 2024-03-24 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2024-03-23 21:04:56,928 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240324/06/ 2024-03-23 21:08:29,055 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240324/00/ 2024-03-23 21:08:29,056 INFO [crop_gribs] 2024-03-24 00 GRIBs cropping complete 2024-03-23 21:08:29,058 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-03-23 21:16:22,890 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 21:16:23,289 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-06'], host='localhost') 2024-03-23 21:16:23,295 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 21:16:23,300 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 21:16:24,328 INFO [download_results] running in process 1573823 2024-03-23 21:16:24,329 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 21:16:24,330 INFO [download_results] publishing log messages to tcp://*:5566 2024-03-23 21:16:24,331 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 21:16:24,332 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/06oct19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 21:16:28,497 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 21:16:53,267 INFO [download_results] hindcast 2019-10-06 results files from optimum-hindcast downloaded 2024-03-23 21:16:53,274 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 21:16:53,489 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-06'], host='localhost') 2024-03-23 21:16:54,344 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 22:06:29,198 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 22:06:29,599 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-11'], host='localhost') 2024-03-23 22:06:29,606 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 22:06:29,610 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 22:06:30,598 INFO [download_results] running in process 1574316 2024-03-23 22:06:30,599 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 22:06:30,600 INFO [download_results] publishing log messages to tcp://*:5563 2024-03-23 22:06:30,602 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 22:06:30,603 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/11oct19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 22:06:34,792 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 22:06:59,423 INFO [download_results] hindcast 2019-10-11 results files from optimum-hindcast downloaded 2024-03-23 22:06:59,430 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 22:06:59,645 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-11'], host='localhost') 2024-03-23 22:07:00,542 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 22:56:35,448 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 22:56:39,003 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-16'], host='localhost') 2024-03-23 22:56:39,011 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 22:56:39,016 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 22:56:39,988 INFO [download_results] running in process 1575087 2024-03-23 22:56:39,990 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 22:56:39,991 INFO [download_results] publishing log messages to tcp://*:5563 2024-03-23 22:56:39,992 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 22:56:39,993 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/16oct19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 22:56:44,138 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 22:57:08,878 INFO [download_results] hindcast 2019-10-16 results files from optimum-hindcast downloaded 2024-03-23 22:57:08,886 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 22:57:09,093 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-16'], host='localhost') 2024-03-23 22:57:09,955 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-23 23:46:44,731 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-23 23:46:45,105 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-21'], host='localhost') 2024-03-23 23:46:45,112 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 23:46:45,117 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-23 23:46:46,094 INFO [download_results] running in process 1575696 2024-03-23 23:46:46,096 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-23 23:46:46,097 INFO [download_results] publishing log messages to tcp://*:5563 2024-03-23 23:46:46,098 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-23 23:46:46,099 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/21oct19 to /ocean/sallen/timbucktwo/rivers 2024-03-23 23:46:50,270 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-23 23:47:14,990 INFO [download_results] hindcast 2019-10-21 results files from optimum-hindcast downloaded 2024-03-23 23:47:14,997 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-23 23:47:15,205 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-21'], host='localhost') 2024-03-23 23:47:16,088 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-24 00:41:51,357 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-24 00:41:51,814 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-26'], host='localhost') 2024-03-24 00:41:51,820 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 00:41:51,825 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 00:41:52,814 INFO [download_results] running in process 1576307 2024-03-24 00:41:52,815 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 00:41:52,816 INFO [download_results] publishing log messages to tcp://*:5565 2024-03-24 00:41:52,817 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-24 00:41:52,818 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/26oct19 to /ocean/sallen/timbucktwo/rivers 2024-03-24 00:41:56,860 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-24 00:42:24,463 INFO [download_results] hindcast 2019-10-26 results files from optimum-hindcast downloaded 2024-03-24 00:42:24,471 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-24 00:42:24,683 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-26'], host='localhost') 2024-03-24 00:42:25,559 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-24 00:42:25,770 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-oct', 'robot.graham'], host='localhost') 2024-03-24 00:42:26,558 INFO [archive_tarball] running in process 1576357 2024-03-24 00:42:26,560 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 00:42:26,561 INFO [archive_tarball] publishing log messages to tcp://*:5563 2024-03-24 00:42:26,562 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-03-24 00:42:26,563 INFO [archive_tarball] creating /ocean/dlatorne/rivers-oct19.tar from /ocean/sallen/timbucktwo/rivers/*oct19/ 2024-03-24 00:45:29,302 INFO [archive_tarball] creating /ocean/dlatorne/rivers-oct19.index from /ocean/dlatorne/rivers-oct19.tar 2024-03-24 00:45:29,361 INFO [archive_tarball] rsync-ing /ocean/dlatorne/rivers-oct19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/rivers/ 2024-03-24 01:07:50,280 INFO [archive_tarball] hindcast *oct19 results files archived to robot.graham 2024-03-24 01:07:50,283 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-03-24 01:31:57,671 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-24 01:31:58,077 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-01'], host='localhost') 2024-03-24 01:31:58,084 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 01:31:58,089 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 01:31:59,063 INFO [download_results] running in process 1576813 2024-03-24 01:31:59,064 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 01:31:59,065 INFO [download_results] publishing log messages to tcp://*:5565 2024-03-24 01:31:59,066 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-24 01:31:59,067 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/01nov19 to /ocean/sallen/timbucktwo/rivers 2024-03-24 01:32:03,170 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-24 01:32:27,818 INFO [download_results] hindcast 2019-11-01 results files from optimum-hindcast downloaded 2024-03-24 01:32:27,826 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-24 01:32:28,034 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-01'], host='localhost') 2024-03-24 01:32:28,965 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-24 02:22:03,962 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-24 02:22:04,372 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-06'], host='localhost') 2024-03-24 02:22:04,379 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 02:22:04,384 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 02:22:05,357 INFO [download_results] running in process 1576943 2024-03-24 02:22:05,358 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 02:22:05,359 INFO [download_results] publishing log messages to tcp://*:5565 2024-03-24 02:22:05,360 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-24 02:22:05,361 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/06nov19 to /ocean/sallen/timbucktwo/rivers 2024-03-24 02:22:09,439 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-24 02:22:33,989 INFO [download_results] hindcast 2019-11-06 results files from optimum-hindcast downloaded 2024-03-24 02:22:33,997 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-24 02:22:34,214 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-06'], host='localhost') 2024-03-24 02:22:35,029 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-24 03:03:09,675 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240324/06/ 2024-03-24 03:03:09,676 INFO [collect_weather] 2024-03-24 2.5km weather forecast 06 collection complete 2024-03-24 03:03:09,679 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-03-24 03:03:10,081 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,086 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,090 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,095 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,105 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,109 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,114 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,118 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,132 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,137 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,142 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,154 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,165 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,170 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-03-23'], host='localhost') 2024-03-24 03:03:10,177 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2024-03-24 03:03:10,188 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2024-03-24 03:03:10,199 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2024-03-24 03:03:10,205 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2024-03-24 03:03:10,220 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2024-03-24 03:03:10,247 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2024-03-24 03:03:12,155 INFO [collect_weather] running in process 1577279 2024-03-24 03:03:12,157 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:12,158 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-03-24 03:03:12,159 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:12,170 INFO [collect_NeahBay_ssh] running in process 1577278 2024-03-24 03:03:12,172 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:12,174 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563 2024-03-24 03:03:12,178 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:12,179 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2024-03-24 03:03:12,180 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-03-24 2024-03-24 03:03:13,575 INFO [collect_river_data] running in process 1577261 2024-03-24 03:03:13,576 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,577 INFO [collect_river_data] publishing log messages to tcp://*:5565 2024-03-24 03:03:13,578 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,579 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-03-23 2024-03-24 03:03:13,638 INFO [collect_river_data] running in process 1577270 2024-03-24 03:03:13,639 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,640 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-03-24 03:03:13,640 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,641 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-03-23 2024-03-24 03:03:13,664 INFO [collect_river_data] running in process 1577259 2024-03-24 03:03:13,667 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,667 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-03-24 03:03:13,668 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,669 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-03-23 2024-03-24 03:03:13,671 INFO [collect_river_data] running in process 1577273 2024-03-24 03:03:13,672 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,672 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-03-24 03:03:13,674 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,675 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-03-23 2024-03-24 03:03:13,696 INFO [collect_river_data] running in process 1577264 2024-03-24 03:03:13,697 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,698 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-03-24 03:03:13,699 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,700 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-03-23 2024-03-24 03:03:13,718 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2024-03-24 03:03:13,719 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-03-23 completed 2024-03-24 03:03:13,720 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:13,732 INFO [collect_river_data] running in process 1577272 2024-03-24 03:03:13,733 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,734 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2024-03-24 03:03:13,735 INFO [collect_river_data] publishing log messages to tcp://*:5571 2024-03-24 03:03:13,736 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-03-23 completed 2024-03-24 03:03:13,737 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,738 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-03-23 2024-03-24 03:03:13,744 INFO [collect_river_data] running in process 1577267 2024-03-24 03:03:13,745 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,746 INFO [collect_river_data] publishing log messages to tcp://*:5572 2024-03-24 03:03:13,747 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,748 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-03-23 2024-03-24 03:03:13,761 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2024-03-24 03:03:13,762 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-03-23 completed 2024-03-24 03:03:13,797 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2024-03-24 03:03:13,797 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-03-23 completed 2024-03-24 03:03:13,805 INFO [collect_river_data] running in process 1577269 2024-03-24 03:03:13,806 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,808 INFO [collect_river_data] publishing log messages to tcp://*:5573 2024-03-24 03:03:13,809 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,810 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-03-23 2024-03-24 03:03:13,811 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-03-23 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2024-03-24 03:03:13,812 INFO [collect_river_data] ECCC Capilano river data collection for 2024-03-23 completed 2024-03-24 03:03:13,813 INFO [collect_river_data] running in process 1577268 2024-03-24 03:03:13,814 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,815 INFO [collect_river_data] publishing log messages to tcp://*:5574 2024-03-24 03:03:13,817 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,818 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-03-23 2024-03-24 03:03:13,824 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/RobertsCreek_flow 2024-03-24 03:03:13,825 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-03-23 completed 2024-03-24 03:03:13,833 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-03-23 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2024-03-24 03:03:13,834 INFO [collect_river_data] ECCC Fraser river data collection for 2024-03-23 completed 2024-03-24 03:03:13,843 INFO [collect_river_data] running in process 1577262 2024-03-24 03:03:13,844 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,846 INFO [collect_river_data] publishing log messages to tcp://*:5575 2024-03-24 03:03:13,846 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,848 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-03-23 2024-03-24 03:03:13,854 INFO [collect_NeahBay_ssh] 2024-03-24T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2024-03-24 03:03:13,857 INFO [collect_river_data] running in process 1577271 2024-03-24 03:03:13,858 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,859 INFO [collect_river_data] publishing log messages to tcp://*:5576 2024-03-24 03:03:13,860 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,861 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-03-23 2024-03-24 03:03:13,861 INFO [collect_river_data] running in process 1577265 2024-03-24 03:03:13,862 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,863 INFO [collect_river_data] publishing log messages to tcp://*:5577 2024-03-24 03:03:13,865 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,866 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-03-23 2024-03-24 03:03:13,868 INFO [collect_river_data] running in process 1577266 2024-03-24 03:03:13,869 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,870 INFO [collect_river_data] publishing log messages to tcp://*:5578 2024-03-24 03:03:13,871 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,872 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-03-23 2024-03-24 03:03:13,879 INFO [collect_river_data] running in process 1577263 2024-03-24 03:03:13,880 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:13,881 INFO [collect_river_data] publishing log messages to tcp://*:5579 2024-03-24 03:03:13,882 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:13,883 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-03-23 2024-03-24 03:03:13,894 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2024-03-24 03:03:13,896 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-03-23 completed 2024-03-24 03:03:13,909 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2024-03-24 03:03:13,910 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-03-23 completed 2024-03-24 03:03:13,926 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2024-03-24 03:03:13,927 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-03-23 completed 2024-03-24 03:03:13,954 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2024-03-24 03:03:13,955 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-03-23 completed 2024-03-24 03:03:13,959 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2024-03-24 03:03:13,960 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-03-23 completed 2024-03-24 03:03:13,966 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-03-23 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2024-03-24 03:03:13,967 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-03-23 completed 2024-03-24 03:03:13,981 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:13,991 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-03-23 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2024-03-24 03:03:13,992 INFO [collect_river_data] ECCC Englishman river data collection for 2024-03-23 completed 2024-03-24 03:03:14,164 INFO [crop_gribs] running in process 1577280 2024-03-24 03:03:14,165 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:14,166 INFO [crop_gribs] publishing log messages to tcp://*:5590 2024-03-24 03:03:14,167 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:14,167 INFO [crop_gribs] cropping 2024-03-24 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2024-03-24 03:03:14,175 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240324/12/ 2024-03-24 03:03:14,186 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:14,393 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:14,498 INFO [get_onc_ctd] running in process 1577276 2024-03-24 03:03:14,499 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:14,500 INFO [get_onc_ctd] publishing log messages to tcp://*:5565 2024-03-24 03:03:14,501 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:14,502 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-03-23 2024-03-24 03:03:14,508 INFO [get_onc_ctd] running in process 1577275 2024-03-24 03:03:14,509 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:14,510 INFO [get_onc_ctd] publishing log messages to tcp://*:5567 2024-03-24 03:03:14,511 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:14,511 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-03-23 2024-03-24 03:03:14,526 INFO [get_onc_ferry] running in process 1577277 2024-03-24 03:03:14,528 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:14,528 INFO [get_onc_ferry] publishing log messages to tcp://*:5591 2024-03-24 03:03:14,529 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:14,530 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-03-23 2024-03-24 03:03:14,578 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:14,765 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:14,961 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:15,155 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-03-24 03:03:15,340 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-03-24'], host='localhost') 2024-03-24 03:03:15,345 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:15,439 INFO [get_onc_ferry] requesting ONC TWDP.N2 NAV data for 2024-03-23 2024-03-24 03:03:15,518 ERROR [get_onc_ferry] request for ONC TWDP.N2 NAV data for 2024-03-23 failed: 400 Client Error: for url: https://data.oceannetworks.ca/api/scalardata?method=getByLocation&token=8efe44eb-0fb5-4413-bba3-ae3dbe0fbb1d&locationCode=TWDP.N2&deviceCategoryCode=NAV&sensorCategoryCodes=longitude%2Clatitude&dateFrom=2024-03-23T00:00:00.000Z&dateTo=2024-03-23T23:59:00.000Z&resampleType=avg&resamplePeriod=1 2024-03-24 03:03:15,660 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:15,889 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:15,931 CRITICAL [get_onc_ferry] 2024-03-23 ONC TWDP ferry data file creation failed 2024-03-24 03:03:16,098 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:16,291 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:16,483 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:16,685 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-03-24 03:03:18,288 INFO [make_ssh_files] running in process 1577679 2024-03-24 03:03:18,289 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:18,290 INFO [make_ssh_files] publishing log messages to tcp://*:5563 2024-03-24 03:03:18,291 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:18,292 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-03-24 NOAA Neah Bay 00Z observation and forecast values 2024-03-24 03:03:23,057 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d19.nc 2024-03-24 03:03:23,222 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d20.nc 2024-03-24 03:03:23,391 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d21.nc 2024-03-24 03:03:23,556 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d22.nc 2024-03-24 03:03:23,723 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m03d23.nc 2024-03-24 03:03:23,941 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m03d24.nc 2024-03-24 03:03:24,128 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m03d25.nc 2024-03-24 03:03:24,295 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m03d26.nc 2024-03-24 03:03:24,458 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m03d27.nc 2024-03-24 03:03:24,591 INFO [make_ssh_files] sea surface height boundary file for 2024-03-24 forecast2 run created 2024-03-24 03:03:24,593 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-03-24 03:03:24,802 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-03-24 03:03:24,807 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-03-24 03:03:25,851 INFO [make_runoff_file] running in process 1577724 2024-03-24 03:03:25,852 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:25,853 INFO [make_runoff_file] publishing log messages to tcp://*:5563 2024-03-24 03:03:25,855 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:25,923 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-03-24 03:03:26,032 INFO [make_runoff_file] 2024-03-24 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-03-24 03:03:26,034 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-03-24 03:03:26,567 INFO [make_v202111_runoff_file] running in process 1577722 2024-03-24 03:03:26,569 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:26,569 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5563 2024-03-24 03:03:26,570 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:26,571 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-03-23 2024-03-24 03:03:27,618 ERROR [make_v202111_runoff_file] no 2024-03-23 discharge obs for Skagit_MountVernon: patching 2024-03-24 03:03:27,924 ERROR [make_v202111_runoff_file] no 2024-03-23 discharge obs for Snohomish_Monroe: patching 2024-03-24 03:03:28,154 ERROR [make_v202111_runoff_file] no 2024-03-23 discharge obs for Nisqually_McKenna: patching 2024-03-24 03:03:28,252 ERROR [make_v202111_runoff_file] no 2024-03-23 discharge obs for Greenwater_Greenwater: patching 2024-03-24 03:03:29,520 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-03-23: /results/forcing/rivers/R202108Dailies_y2024m03d23.nc 2024-03-24 03:03:29,521 INFO [make_v202111_runoff_file] 2024-03-23 runoff file creation completed 2024-03-24 03:03:29,523 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-03-24 03:03:34,326 INFO [get_onc_ctd] 2024-03-23 ONC SEVIP CTD T&S file created 2024-03-24 03:03:34,328 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2024-03-24 03:03:34,532 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2024-03-24 03:03:35,339 INFO [ping_erddap] running in process 1577802 2024-03-24 03:03:35,341 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:03:35,342 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-03-24 03:03:35,343 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:03:35,344 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2024-03-24 03:03:35,346 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-03-24 03:03:47,402 CRITICAL [get_onc_ctd] 2024-03-23 ONC SCVIP CTD T&S file creation failed 2024-03-24 03:07:09,848 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-24 03:07:10,305 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-11'], host='localhost') 2024-03-24 03:07:10,311 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 03:07:10,316 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 03:07:11,267 INFO [download_results] running in process 1577816 2024-03-24 03:07:11,268 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:07:11,269 INFO [download_results] publishing log messages to tcp://*:5563 2024-03-24 03:07:11,270 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:07:11,271 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/11nov19 to /ocean/sallen/timbucktwo/rivers 2024-03-24 03:07:15,485 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-24 03:07:39,951 INFO [download_results] hindcast 2019-11-11 results files from optimum-hindcast downloaded 2024-03-24 03:07:39,958 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-24 03:07:40,172 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-11'], host='localhost') 2024-03-24 03:07:41,048 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-24 03:10:06,035 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240324/06/ 2024-03-24 03:10:06,036 INFO [crop_gribs] 2024-03-24 06 GRIBs cropping complete 2024-03-24 03:10:06,038 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-03-24 03:10:06,467 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2024-03-24 03:10:08,322 INFO [grib_to_netcdf] running in process 1577944 2024-03-24 03:10:08,324 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:10:08,325 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563 2024-03-24 03:10:08,326 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:10:08,327 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-03-24 forecast2 run 2024-03-24 03:10:19,818 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m03d25.nc 2024-03-24 03:10:23,430 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m03d26.nc 2024-03-24 03:10:23,440 INFO [grib_to_netcdf] 2024-03-24 NEMO-atmos forcing file for forecast2 created 2024-03-24 03:10:23,443 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-03-24 03:10:23,834 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2024-03-24 03:10:23,835 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2024-03-24 03:10:23,836 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2024-03-24 03:10:23,837 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2024-03-24 03:10:24,566 INFO [upload_forcing] running in process 1578029 2024-03-24 03:10:24,567 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:10:24,568 INFO [upload_forcing] publishing log messages to tcp://*:5563 2024-03-24 03:10:24,569 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:10:24,592 INFO [upload_forcing] running in process 1578032 2024-03-24 03:10:24,594 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:10:24,595 INFO [upload_forcing] publishing log messages to tcp://*:5564 2024-03-24 03:10:24,596 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:10:24,597 INFO [upload_forcing] running in process 1578031 2024-03-24 03:10:24,598 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:10:24,599 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-03-24 03:10:24,599 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:10:24,621 INFO [upload_forcing] running in process 1578033 2024-03-24 03:10:24,622 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:10:24,623 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-03-24 03:10:24,624 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:10:25,840 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m03d23.nc 2024-03-24 03:10:25,875 INFO [upload_forcing] forecast2 2024-03-24 forcing files upload to optimum-hindcast completed 2024-03-24 03:10:25,879 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-24 03:10:26,367 INFO [upload_forcing] forecast2 2024-03-24 forcing files upload to orcinus-nowcast-agrif completed 2024-03-24 03:10:26,371 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-24 03:10:27,103 INFO [upload_forcing] forecast2 2024-03-24 forcing files upload to arbutus.cloud-nowcast completed 2024-03-24 03:10:27,108 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-24 03:10:27,304 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-03-24'], host='localhost') 2024-03-24 03:10:28,149 INFO [make_forcing_links] running in process 1578060 2024-03-24 03:10:28,150 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:10:28,151 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-03-24 03:10:28,152 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:10:29,282 INFO [make_forcing_links] forecast2 2024-03-24 forcing file links on arbutus.cloud-nowcast created 2024-03-24 03:10:29,286 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-03-24 03:10:29,507 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-03-24'], host='arbutus.cloud-nowcast') 2024-03-24 03:10:31,556 INFO [run_NEMO] running in process 21071 2024-03-24 03:10:31,557 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:10:31,558 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-03-24 03:10:32,467 INFO [upload_forcing] forecast2 2024-03-24 forcing files upload to robot.graham completed 2024-03-24 03:10:32,473 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-03-24 03:10:41,571 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:10:52,146 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/23mar24forecast2_2024-03-24T031051.329233-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-03-24 03:10:52,167 INFO [run_NEMO] forecast2 NEMO run for 2024-03-24 on arbutus.cloud-nowcast started 2024-03-24 03:10:52,171 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-03-24 03:10:52,389 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-03-24 03:10:54,335 INFO [watch_NEMO] running in process 21323 2024-03-24 03:10:54,336 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:10:54,338 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-03-24 03:10:54,339 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:10:54,408 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-03-24 03:15:54,409 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17543960 = 2024-03-25 04:52:40 UTC, 16.3% complete 2024-03-24 03:20:54,410 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17544451 = 2024-03-25 10:20:00 UTC, 34.5% complete 2024-03-24 03:25:54,412 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17544929 = 2024-03-25 15:38:40 UTC, 52.2% complete 2024-03-24 03:30:54,414 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17545409 = 2024-03-25 20:58:40 UTC, 70.0% complete 2024-03-24 03:35:54,414 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17545890 = 2024-03-26 02:19:20 UTC, 87.8% complete 2024-03-24 03:40:54,418 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2024-03-24 03:40:54,422 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-03-24 03:40:54,885 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-03-24'], host='arbutus.cloud-nowcast') 2024-03-24 03:40:54,892 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-03-24'], host='arbutus.cloud-nowcast') 2024-03-24 03:40:54,896 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-03-23'], host='localhost') 2024-03-24 03:40:55,983 INFO [download_results] running in process 1578297 2024-03-24 03:40:55,984 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:40:55,985 INFO [download_results] publishing log messages to tcp://*:5563 2024-03-24 03:40:55,986 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:40:55,988 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/23mar24 to /results/SalishSea/forecast2.202111 2024-03-24 03:41:01,877 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-03-24 03:42:54,319 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-03-24 03:42:54,537 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-03-24'], host='arbutus.cloud-nowcast') 2024-03-24 03:42:57,033 INFO [run_ww3] forecast2 WaveWatch3 run for 2024-03-24 on arbutus.cloud-nowcast started 2024-03-24 03:42:57,148 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-03-24 03:42:57,362 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-03-24 03:42:59,891 INFO [watch_ww3] running in process 21862 2024-03-24 03:42:59,891 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:42:59,892 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-03-24 03:42:59,893 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:43:00,154 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-03-24 03:48:00,175 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete 2024-03-24 03:53:00,362 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 192, 35.6% complete 2024-03-24 03:57:15,990 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-03-24 03:57:16,482 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-16'], host='localhost') 2024-03-24 03:57:16,490 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 03:57:16,494 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-03-24 03:57:17,512 INFO [download_results] running in process 1578533 2024-03-24 03:57:17,513 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 03:57:17,514 INFO [download_results] publishing log messages to tcp://*:5564 2024-03-24 03:57:17,515 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-03-24 03:57:17,516 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/rivers/16nov19 to /ocean/sallen/timbucktwo/rivers 2024-03-24 03:57:21,744 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-03-24 03:57:46,636 INFO [download_results] hindcast 2019-11-16 results files from optimum-hindcast downloaded 2024-03-24 03:57:46,644 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-24 03:57:46,859 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-16'], host='localhost') 2024-03-24 03:57:47,804 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-03-24 03:58:00,147 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 297, 55.0% complete 2024-03-24 04:03:00,414 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 396, 73.3% complete 2024-03-24 04:05:33,732 INFO [download_results] forecast2 2024-03-23 results files from arbutus.cloud-nowcast downloaded 2024-03-24 04:05:33,736 INFO [manager] checklist updated with [results files] items from download_results worker 2024-03-24 04:05:33,946 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-03-23'], host='localhost') 2024-03-24 04:05:35,843 INFO [make_CHS_currents_file] running in process 1578642 2024-03-24 04:05:35,845 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:05:35,846 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-03-24 04:05:35,847 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:06:10,514 INFO [make_CHS_currents_file] Made CHS currents file for 2024-03-23 for forecast2 2024-03-24 04:06:10,517 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-03-24 04:06:10,731 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-03-23'], host='localhost') 2024-03-24 04:06:11,591 INFO [update_forecast_datasets] running in process 1578666 2024-03-24 04:06:11,592 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:06:11,594 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-03-24 04:06:11,595 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:06:11,596 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/23mar24 run 2024-03-24 04:08:00,071 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 495, 91.7% complete 2024-03-24 04:10:25,119 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/23mar24 run: /results/SalishSea/rolling-forecasts/nemo 2024-03-24 04:10:25,120 INFO [update_forecast_datasets] nemo 2024-03-23 forecast2 rolling forecast datasets updated 2024-03-24 04:10:25,122 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-03-24 04:10:25,333 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-03-24 04:10:25,338 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-03-23'], host='localhost') 2024-03-24 04:10:25,343 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-03-23'], host='localhost') 2024-03-24 04:10:26,300 INFO [ping_erddap] running in process 1578743 2024-03-24 04:10:26,301 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:10:26,302 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-03-24 04:10:26,304 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:10:26,314 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-03-24 04:10:26,316 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-03-24 04:10:27,524 INFO [make_surface_current_tiles] running in process 1578746 2024-03-24 04:10:27,526 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:10:27,528 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2024-03-24 04:10:27,529 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:10:27,575 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-03-23 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/23mar24 2024-03-24 04:10:30,174 INFO [make_plots] running in process 1578745 2024-03-24 04:10:30,175 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:10:30,176 INFO [make_plots] publishing log messages to tcp://*:5564 2024-03-24 04:10:30,178 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:10:31,128 INFO [make_plots] preparing render list for 2024-03-23 NEMO forecast2 publish figures 2024-03-24 04:10:31,300 INFO [make_plots] added Website_thumbnail figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,301 INFO [make_plots] added Threshold_website figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,302 INFO [make_plots] added PA_tidal_predictions figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,303 INFO [make_plots] added Boundary Bay figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,304 INFO [make_plots] added Campbell River figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,305 INFO [make_plots] added Cherry Point figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,306 INFO [make_plots] added Friday Harbor figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,307 INFO [make_plots] added Halfmoon Bay figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,308 INFO [make_plots] added Nanaimo figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,309 INFO [make_plots] added Neah Bay figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,309 INFO [make_plots] added New Westminster figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,310 INFO [make_plots] added Patricia Bay figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,310 INFO [make_plots] added Point Atkinson figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,311 INFO [make_plots] added Port Renfrew figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,312 INFO [make_plots] added Sand Heads figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,313 INFO [make_plots] added Sandy Cove figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,313 INFO [make_plots] added Squamish figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,314 INFO [make_plots] added Victoria figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,315 INFO [make_plots] added Woodwards Landing figure to 2024-03-23 NEMO forecast2 publish render list 2024-03-24 04:10:31,317 INFO [make_plots] starting to render nemo forecast2 publish 23mar24 figures 2024-03-24 04:13:00,069 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2024-03-24 04:13:00,075 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-03-24 04:13:00,646 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-03-24'], host='localhost') 2024-03-24 04:13:01,920 INFO [download_wwatch3_results] running in process 1579266 2024-03-24 04:13:01,921 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:13:01,923 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566 2024-03-24 04:13:01,924 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:15:21,542 INFO [download_wwatch3_results] forecast2 2024-03-24 results files from arbutus.cloud-nowcast downloaded 2024-03-24 04:15:21,547 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-03-24 04:15:21,992 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-03-24'], host='localhost') 2024-03-24 04:15:23,395 INFO [update_forecast_datasets] running in process 1579630 2024-03-24 04:15:23,396 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:15:23,398 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566 2024-03-24 04:15:23,399 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:15:23,400 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/24mar24 run 2024-03-24 04:15:23,421 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/24mar24 run 2024-03-24 04:16:04,550 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/24mar24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-03-24 04:16:04,551 INFO [update_forecast_datasets] wwatch3 2024-03-24 forecast2 rolling forecast datasets updated 2024-03-24 04:16:04,554 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-03-24 04:16:05,031 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-03-24 04:16:06,265 INFO [ping_erddap] running in process 1579766 2024-03-24 04:16:06,266 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:16:06,268 INFO [ping_erddap] publishing log messages to tcp://*:5566 2024-03-24 04:16:06,269 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:16:06,271 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-03-24 04:16:06,273 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-03-24 04:16:06,721 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-03-24'], host='localhost') 2024-03-24 04:16:11,850 INFO [make_plots] running in process 1579772 2024-03-24 04:16:11,851 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:16:11,852 INFO [make_plots] publishing log messages to tcp://*:5566 2024-03-24 04:16:11,853 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:16:11,854 INFO [make_plots] preparing render list for 2024-03-24 WaveWatch3 forecast2 publish figures 2024-03-24 04:16:11,855 INFO [make_plots] added Halibut Bank figure to 2024-03-24 WaveWatch3 forecast2 publish render list 2024-03-24 04:16:11,856 INFO [make_plots] added Sentry Shoal figure to 2024-03-24 WaveWatch3 forecast2 publish render list 2024-03-24 04:16:11,856 INFO [make_plots] starting to render wwatch3 forecast2 publish 24mar24 figures 2024-03-24 04:16:12,179 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-03-24 04:16:23,887 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-03-24 04:16:34,009 INFO [make_plots] finished rendering wwatch3 forecast2 publish 24mar24 figures 2024-03-24 04:16:34,011 INFO [make_plots] wwatch3 publish plots for 2024-03-24 forecast2 completed 2024-03-24 04:16:34,013 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-03-24 04:25:30,775 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/23mar24 2024-03-24 04:28:50,824 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-03-24 04:28:50,825 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-03-24 04:29:06,984 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/23mar24 2024-03-24 04:29:06,993 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-03-23 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/23mar24 2024-03-24 04:29:06,994 INFO [make_surface_current_tiles] surface current tile figures for 2024-03-23 forecast2 completed 2024-03-24 04:29:07,311 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-03-24 04:29:26,510 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:31:14,939 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:31:15,348 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:31:51,093 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:33:02,773 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:33:40,058 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:34:16,026 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:34:52,113 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:35:27,931 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:36:04,488 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:36:40,814 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:36:41,230 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:37:16,467 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:37:52,854 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-03-19 00:00:00Z to 2024-03-26 05:55:00Z 2024-03-24 04:38:29,412 INFO [make_plots] finished rendering nemo forecast2 publish 23mar24 figures 2024-03-24 04:38:29,414 INFO [make_plots] nemo publish plots for 2024-03-23 forecast2 completed 2024-03-24 04:38:29,422 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-03-24 04:38:29,720 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-03-23'], host='localhost') 2024-03-24 04:38:32,198 INFO [make_feeds] running in process 1581163 2024-03-24 04:38:32,200 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:38:32,201 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-03-24 04:38:32,202 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:38:59,133 INFO [make_feeds] ATOM feeds for 2024-03-23 forecast2 run completed 2024-03-24 04:38:59,136 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-03-24 04:38:59,484 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2024-03-24 04:39:00,209 INFO [clear_checklist] running in process 1581186 2024-03-24 04:39:00,211 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:39:00,212 INFO [clear_checklist] publishing log messages to tcp://*:5563 2024-03-24 04:39:00,213 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:39:00,214 INFO [clear_checklist] requesting that manager clear system state checklist 2024-03-24 04:39:00,216 INFO [manager] writing checklist to log file 2024-03-24 04:39:00,266 INFO [manager] checklist cleared 2024-03-24 04:39:00,268 INFO [clear_checklist] nowcast system checklist cleared 2024-03-24 04:39:00,270 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2024-03-24 04:39:01,125 INFO [rotate_logs] running in process 1581191 2024-03-24 04:39:01,127 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-03-24 04:39:01,128 INFO [rotate_logs] publishing log messages to tcp://*:5563 2024-03-24 04:39:01,129 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2024-03-24 04:39:01,129 INFO [rotate_logs] rotating log files