2024-04-17 04:43:12,487 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2024-04-17 04:43:12,494 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2024-04-17 04:43:12,527 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2024-04-17 04:43:12,563 INFO [rotate_logs] log files rotated 2024-04-17 04:43:12,580 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2024-04-17 04:56:36,909 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 04:56:37,156 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-06'], host='localhost') 2024-04-17 04:56:37,162 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 04:56:37,167 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 04:56:38,403 INFO [download_results] running in process 2004687 2024-04-17 04:56:38,404 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 04:56:38,405 INFO [download_results] publishing log messages to tcp://*:5562 2024-04-17 04:56:38,407 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 04:56:38,408 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06feb19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 04:56:42,777 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 04:57:33,399 INFO [download_results] hindcast 2019-02-06 results files from optimum-hindcast downloaded 2024-04-17 04:57:33,415 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 04:57:33,425 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-06'], host='localhost') 2024-04-17 04:57:34,561 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 06:21:47,245 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 06:21:47,577 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-11'], host='localhost') 2024-04-17 06:21:47,585 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 06:21:47,594 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 06:21:49,349 INFO [download_results] running in process 2011317 2024-04-17 06:21:49,350 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 06:21:49,351 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-17 06:21:49,352 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 06:21:49,353 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11feb19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 06:21:53,692 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 06:22:47,866 INFO [download_results] hindcast 2019-02-11 results files from optimum-hindcast downloaded 2024-04-17 06:22:47,880 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 06:22:47,891 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-11'], host='localhost') 2024-04-17 06:22:49,081 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 07:46:58,051 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 07:46:58,253 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-16'], host='localhost') 2024-04-17 07:46:58,259 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 07:46:58,263 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 07:46:59,649 INFO [download_results] running in process 2011688 2024-04-17 07:46:59,651 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 07:46:59,651 INFO [download_results] publishing log messages to tcp://*:5566 2024-04-17 07:46:59,652 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 07:46:59,653 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16feb19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 07:47:03,903 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 07:48:04,633 INFO [download_results] hindcast 2019-02-16 results files from optimum-hindcast downloaded 2024-04-17 07:48:04,648 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 07:48:04,656 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-16'], host='localhost') 2024-04-17 07:48:05,750 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 09:09:05,639 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240417/12/ 2024-04-17 09:09:05,640 INFO [collect_weather] 2024-04-17 2.5km weather forecast 12 collection complete 2024-04-17 09:09:05,644 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-04-17 09:09:05,864 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-04-16'], host='localhost') 2024-04-17 09:09:05,870 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-04-16'], host='localhost') 2024-04-17 09:09:05,882 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-04-16'], host='localhost') 2024-04-17 09:09:05,886 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-04-16'], host='localhost') 2024-04-17 09:09:05,892 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2024-04-17 09:09:05,897 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2024-04-17 09:09:05,903 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2024-04-17 09:09:05,911 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2024-04-17 09:09:05,929 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2024-04-17 09:09:07,544 INFO [collect_weather] running in process 2012380 2024-04-17 09:09:07,545 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:07,546 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-04-17 09:09:07,547 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:07,556 INFO [collect_NeahBay_ssh] running in process 2012378 2024-04-17 09:09:07,557 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:07,558 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5565 2024-04-17 09:09:07,560 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:07,561 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-04-17 2024-04-17 09:09:07,565 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2024-04-17 09:09:07,793 INFO [download_live_ocean] running in process 2012379 2024-04-17 09:09:07,793 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:07,794 INFO [download_live_ocean] publishing log messages to tcp://*:5566 2024-04-17 09:09:07,795 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:07,796 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-04-17 2024-04-17 09:09:08,824 INFO [collect_river_data] running in process 2012372 2024-04-17 09:09:08,825 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:08,826 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-04-17 09:09:08,827 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:08,828 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-04-16 2024-04-17 09:09:08,830 INFO [collect_river_data] running in process 2012374 2024-04-17 09:09:08,831 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:08,832 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-04-17 09:09:08,834 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:08,834 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-04-16 2024-04-17 09:09:08,841 INFO [collect_river_data] running in process 2012375 2024-04-17 09:09:08,842 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:08,844 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-04-17 09:09:08,844 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:08,845 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-04-16 2024-04-17 09:09:08,854 INFO [collect_river_data] running in process 2012376 2024-04-17 09:09:08,855 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:08,856 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-04-17 09:09:08,858 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:08,858 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-04-16 2024-04-17 09:09:09,060 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2024-04-16&endDT=2024-04-16 "HTTP/1.1 200 OK" 2024-04-17 09:09:09,111 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-04-16 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2024-04-17 09:09:09,129 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-04-16 completed 2024-04-17 09:09:09,136 INFO [make_turbidity_file] running in process 2012377 2024-04-17 09:09:09,139 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:09,140 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-17 09:09:09,143 INFO [make_turbidity_file] publishing log messages to tcp://*:5571 2024-04-17 09:09:09,146 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:09,149 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-04-17 2024-04-17 09:09:09,191 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2024-04-16&endDT=2024-04-16 "HTTP/1.1 200 OK" 2024-04-17 09:09:09,192 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2024-04-16&endDT=2024-04-16 "HTTP/1.1 200 OK" 2024-04-17 09:09:09,198 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2024-04-16&endDT=2024-04-16 "HTTP/1.1 200 OK" 2024-04-17 09:09:09,245 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-04-16 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2024-04-17 09:09:09,245 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-04-16 completed 2024-04-17 09:09:09,248 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-17 09:09:09,291 INFO [collect_NeahBay_ssh] 2024-04-17T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2024-04-17 09:09:09,294 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-04-17 09:09:09,301 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 09:09:09,504 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-04-16 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2024-04-17 09:09:09,505 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-04-16 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2024-04-17 09:09:09,506 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-04-16 completed 2024-04-17 09:09:09,507 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-04-16 completed 2024-04-17 09:09:09,511 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-17 09:09:09,522 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-17 09:09:09,738 INFO [download_live_ocean] downloaded 10116786 bytes from https://liveocean.apl.uw.edu/output/f2024.04.17/ubc.nc to /results/forcing/LiveOcean/downloaded/20240417/low_passed_UBC.nc 2024-04-17 09:09:09,744 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2024-04-17 09:09:09,920 INFO [crop_gribs] running in process 2012381 2024-04-17 09:09:09,921 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:09,922 INFO [crop_gribs] publishing log messages to tcp://*:5565 2024-04-17 09:09:09,923 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:09,924 INFO [crop_gribs] cropping 2024-04-17 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2024-04-17 09:09:09,936 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240417/18/ 2024-04-17 09:09:10,806 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240417/low_passed_UBC.nc 2024-04-17 09:09:10,807 INFO [download_live_ocean] 2024-04-17 Live Ocean file for Salish Sea western boundary downloaded 2024-04-17 09:09:10,810 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2024-04-17 09:09:11,070 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-04-17'], host='localhost') 2024-04-17 09:09:12,461 INFO [make_turbidity_file] 2024-04-17 Fraser River turbidity file creation complete 2024-04-17 09:09:12,464 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2024-04-17 09:09:14,082 INFO [make_ssh_files] running in process 2012524 2024-04-17 09:09:14,083 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:14,084 INFO [make_ssh_files] publishing log messages to tcp://*:5566 2024-04-17 09:09:14,085 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:14,086 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-04-17 NOAA Neah Bay 06Z observation and forecast values 2024-04-17 09:09:14,090 INFO [make_live_ocean_files] running in process 2012546 2024-04-17 09:09:14,091 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:14,092 INFO [make_live_ocean_files] publishing log messages to tcp://*:5567 2024-04-17 09:09:14,094 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:14,096 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-04-17 Live Ocean run 2024-04-17 09:09:18,994 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d13.nc 2024-04-17 09:09:19,175 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d14.nc 2024-04-17 09:09:19,354 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d15.nc 2024-04-17 09:09:19,564 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d16.nc 2024-04-17 09:09:19,757 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc 2024-04-17 09:09:19,916 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d18.nc 2024-04-17 09:09:20,085 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d19.nc 2024-04-17 09:09:20,249 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d20.nc 2024-04-17 09:09:20,414 INFO [make_ssh_files] sea surface height boundary file for 2024-04-17 nowcast run created 2024-04-17 09:09:20,418 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-04-17 09:09:20,431 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-04-17 09:09:20,436 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-04-17 09:09:21,784 INFO [make_runoff_file] running in process 2012591 2024-04-17 09:09:21,784 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:21,785 INFO [make_runoff_file] publishing log messages to tcp://*:5566 2024-04-17 09:09:21,786 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:22,051 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-04-17 09:09:22,246 INFO [make_runoff_file] 2024-04-17 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-04-17 09:09:22,248 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-04-17 09:09:22,592 INFO [make_v202111_runoff_file] running in process 2012589 2024-04-17 09:09:22,593 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:09:22,594 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5566 2024-04-17 09:09:22,595 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:09:22,596 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-16 2024-04-17 09:09:25,787 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-16: /results/forcing/rivers/R202108Dailies_y2024m04d16.nc 2024-04-17 09:09:25,788 INFO [make_v202111_runoff_file] 2024-04-16 runoff file creation completed 2024-04-17 09:09:25,790 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-04-17 09:10:05,235 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: / 2024-04-17 09:10:05,235 INFO [make_live_ocean_files] 2024-04-17 Live Ocean western boundary conditions files created 2024-04-17 09:10:05,238 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2024-04-17 09:12:08,218 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 09:12:08,440 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-21'], host='localhost') 2024-04-17 09:12:08,453 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 09:12:08,463 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 09:12:10,096 INFO [download_results] running in process 2012634 2024-04-17 09:12:10,098 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:12:10,099 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-17 09:12:10,100 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:12:10,101 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21feb19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 09:12:14,126 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 09:13:07,267 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240417/12/ 2024-04-17 09:13:07,276 INFO [crop_gribs] 2024-04-17 12 GRIBs cropping complete 2024-04-17 09:13:07,295 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-04-17 09:13:07,552 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2024-04-17 09:13:10,234 INFO [grib_to_netcdf] running in process 2012674 2024-04-17 09:13:10,235 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:13:10,236 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563 2024-04-17 09:13:10,237 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:13:10,238 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-17 nowcast and forecast runs 2024-04-17 09:13:27,254 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m04d17.nc 2024-04-17 09:13:39,660 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d18.nc 2024-04-17 09:13:46,098 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d19.nc 2024-04-17 09:13:46,115 INFO [grib_to_netcdf] 2024-04-17 NEMO-atmos forcing file for nowcast+ created 2024-04-17 09:13:46,122 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-04-17 09:13:46,129 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2024-04-17 09:13:46,133 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2024-04-17 09:13:46,138 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2024-04-17 09:13:46,142 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2024-04-17 09:13:46,147 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2024-04-17 09:13:47,383 INFO [ping_erddap] running in process 2012718 2024-04-17 09:13:47,384 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:13:47,385 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-04-17 09:13:47,386 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:13:47,388 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2024-04-17 09:13:47,390 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-17 09:13:47,656 INFO [upload_forcing] running in process 2012716 2024-04-17 09:13:47,657 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:13:47,659 INFO [upload_forcing] publishing log messages to tcp://*:5563 2024-04-17 09:13:47,660 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:13:47,662 INFO [upload_forcing] running in process 2012717 2024-04-17 09:13:47,664 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:13:47,665 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-04-17 09:13:47,666 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:13:47,688 INFO [upload_forcing] running in process 2012713 2024-04-17 09:13:47,689 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:13:47,689 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-04-17 09:13:47,691 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:13:47,707 INFO [upload_forcing] running in process 2012715 2024-04-17 09:13:47,708 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:13:47,710 INFO [upload_forcing] publishing log messages to tcp://*:5569 2024-04-17 09:13:47,711 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:13:50,380 INFO [upload_forcing] nowcast+ 2024-04-17 forcing files upload to optimum-hindcast completed 2024-04-17 09:13:50,394 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-17 09:13:50,525 INFO [upload_forcing] nowcast+ 2024-04-17 forcing files upload to orcinus-nowcast-agrif completed 2024-04-17 09:13:50,529 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-17 09:13:59,033 INFO [upload_forcing] nowcast+ 2024-04-17 forcing files upload to robot.graham completed 2024-04-17 09:13:59,038 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-17 09:13:59,433 INFO [upload_forcing] nowcast+ 2024-04-17 forcing files upload to arbutus.cloud-nowcast completed 2024-04-17 09:13:59,438 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-17 09:13:59,450 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 09:14:00,700 INFO [make_forcing_links] running in process 2012752 2024-04-17 09:14:00,701 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:14:00,726 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-04-17 09:14:00,744 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:14:01,894 INFO [make_forcing_links] nowcast+ 2024-04-17 forcing file links on arbutus.cloud-nowcast created 2024-04-17 09:14:01,921 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-17 09:14:01,923 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-17'], host='arbutus.cloud-nowcast') 2024-04-17 09:14:04,039 INFO [run_NEMO] running in process 23959 2024-04-17 09:14:04,041 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:14:04,042 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-04-17 09:14:04,166 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:14:04,902 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/17apr24nowcast_2024-04-17T091404.096227-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-17 09:14:04,923 INFO [run_NEMO] nowcast NEMO run for 2024-04-17 on arbutus.cloud-nowcast started 2024-04-17 09:14:04,928 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-17 09:14:04,943 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-04-17 09:14:06,925 INFO [watch_NEMO] running in process 24213 2024-04-17 09:14:06,926 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:14:06,927 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-17 09:14:06,928 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:14:06,994 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-17 09:14:23,917 INFO [download_results] hindcast 2019-02-21 results files from optimum-hindcast downloaded 2024-04-17 09:14:23,933 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 09:14:23,948 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-21'], host='localhost') 2024-04-17 09:14:25,168 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 09:19:06,996 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17593633 = 2024-04-17 04:48:00 UTC, 20.0% complete 2024-04-17 09:24:06,997 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17594140 = 2024-04-17 10:26:00 UTC, 43.5% complete 2024-04-17 09:29:06,998 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17594625 = 2024-04-17 15:49:20 UTC, 66.0% complete 2024-04-17 09:34:06,999 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17595104 = 2024-04-17 21:08:40 UTC, 88.1% complete 2024-04-17 09:39:07,002 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2024-04-17 09:39:07,007 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-17 09:39:07,216 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 09:39:07,222 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 09:39:08,817 INFO [make_forcing_links] running in process 2013046 2024-04-17 09:39:08,822 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:39:08,833 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-04-17 09:39:08,851 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:39:08,868 INFO [download_results] running in process 2013047 2024-04-17 09:39:08,869 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:39:08,870 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-17 09:39:08,871 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:39:08,872 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/17apr24 to /results/SalishSea/nowcast-blue.202111 2024-04-17 09:39:09,802 INFO [make_forcing_links] ssh 2024-04-17 forcing file links on arbutus.cloud-nowcast created 2024-04-17 09:39:09,807 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-17 09:39:09,821 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-17'], host='arbutus.cloud-nowcast') 2024-04-17 09:39:12,619 INFO [run_NEMO] running in process 26197 2024-04-17 09:39:12,620 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:39:12,621 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-04-17 09:39:12,622 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:39:13,989 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/17apr24forecast_2024-04-17T093912.693321-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-17 09:39:14,009 INFO [run_NEMO] forecast NEMO run for 2024-04-17 on arbutus.cloud-nowcast started 2024-04-17 09:39:14,014 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-17 09:39:14,032 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-04-17 09:39:16,239 INFO [watch_NEMO] running in process 26453 2024-04-17 09:39:16,240 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:39:16,241 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-17 09:39:16,244 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:39:16,436 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-17 09:44:16,426 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17595803 = 2024-04-18 04:54:40 UTC, 13.6% complete 2024-04-17 09:49:16,809 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17596274 = 2024-04-18 10:08:40 UTC, 28.2% complete 2024-04-17 09:54:16,420 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17596764 = 2024-04-18 15:35:20 UTC, 43.3% complete 2024-04-17 09:59:16,427 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17597256 = 2024-04-18 21:03:20 UTC, 58.5% complete 2024-04-17 09:59:21,396 INFO [download_results] nowcast 2024-04-17 results files from arbutus.cloud-nowcast downloaded 2024-04-17 09:59:21,401 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 09:59:21,415 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 09:59:21,420 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-04-16'], host='localhost') 2024-04-17 09:59:21,426 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 09:59:24,061 INFO [make_CHS_currents_file] running in process 2013320 2024-04-17 09:59:24,062 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:59:24,063 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-04-17 09:59:24,064 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:59:28,501 INFO [make_plots] running in process 2013319 2024-04-17 09:59:28,502 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:59:28,503 INFO [make_plots] publishing log messages to tcp://*:5564 2024-04-17 09:59:28,504 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:59:28,549 INFO [make_plots] running in process 2013317 2024-04-17 09:59:28,557 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 09:59:28,564 INFO [make_plots] publishing log messages to tcp://*:5566 2024-04-17 09:59:28,565 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-17 09:59:29,422 INFO [make_plots] preparing render list for 2024-04-16 NEMO nowcast-blue comparison figures 2024-04-17 09:59:29,445 INFO [make_plots] preparing render list for 2024-04-17 NEMO nowcast-blue research figures 2024-04-17 09:59:30,702 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-blue research render list 2024-04-17 09:59:31,928 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-blue research render list 2024-04-17 09:59:31,966 INFO [make_plots] added Currents_sections_and_surface figure to 2024-04-17 NEMO nowcast-blue research render list 2024-04-17 09:59:31,986 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-04-17 NEMO nowcast-blue research render list 2024-04-17 09:59:32,011 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-04-17 NEMO nowcast-blue research render list 2024-04-17 09:59:32,016 INFO [make_plots] starting to render nemo nowcast research 17apr24 figures 2024-04-17 09:59:50,938 INFO [make_plots] added SH_wind figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,939 INFO [make_plots] added Compare_VENUS_East figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,940 INFO [make_plots] added Compare_VENUS_Central figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,941 INFO [make_plots] added Central_ADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,942 INFO [make_plots] added Central_depavADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,944 INFO [make_plots] added Central_timeavADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,945 INFO [make_plots] added East_ADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,946 INFO [make_plots] added East_depavADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,947 INFO [make_plots] added East_timeavADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,947 INFO [make_plots] added ddl_ADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,948 INFO [make_plots] added ddl_depavADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,949 INFO [make_plots] added ddl_timeavADCP figure to 2024-04-16 NEMO nowcast-blue comparison render list 2024-04-17 09:59:50,950 INFO [make_plots] starting to render nemo nowcast comparison 16apr24 figures 2024-04-17 09:59:59,793 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-17 for nowcast 2024-04-17 09:59:59,796 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-04-17 10:00:54,523 INFO [make_plots] finished rendering nemo nowcast research 17apr24 figures 2024-04-17 10:00:54,526 INFO [make_plots] nemo research plots for 2024-04-17 nowcast completed 2024-04-17 10:00:54,541 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-17 10:01:54,515 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-04-17 10:01:55,030 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-04-17 10:01:55,545 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-04-17 10:01:56,063 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-04-17 10:01:56,581 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-04-17 10:01:57,093 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-04-17 10:01:57,607 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-04-17 10:01:58,126 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-04-17 10:01:58,638 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-04-17 10:01:58,639 INFO [make_plots] finished rendering nemo nowcast comparison 16apr24 figures 2024-04-17 10:01:58,744 INFO [make_plots] nemo comparison plots for 2024-04-16 nowcast completed 2024-04-17 10:01:58,748 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-17 10:02:15,397 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 10:02:15,605 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-26'], host='localhost') 2024-04-17 10:02:15,612 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 10:02:15,621 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 10:02:16,844 INFO [download_results] running in process 2013421 2024-04-17 10:02:16,845 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:02:16,846 INFO [download_results] publishing log messages to tcp://*:5563 2024-04-17 10:02:16,847 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:02:16,848 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26feb19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 10:02:20,853 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 10:02:59,412 INFO [download_results] hindcast 2019-02-26 results files from optimum-hindcast downloaded 2024-04-17 10:02:59,423 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 10:02:59,439 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-26'], host='localhost') 2024-04-17 10:03:00,550 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 10:03:00,571 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-feb', 'robot.graham'], host='localhost') 2024-04-17 10:03:01,571 INFO [archive_tarball] running in process 2013464 2024-04-17 10:03:01,593 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:03:01,612 INFO [archive_tarball] publishing log messages to tcp://*:5563 2024-04-17 10:03:01,629 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:03:01,631 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb19.tar from /ocean/sallen/timbucktwo/oxygen/*feb19/ 2024-04-17 10:04:16,420 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17597696 = 2024-04-19 01:56:40 UTC, 72.1% complete 2024-04-17 10:09:16,411 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17598164 = 2024-04-19 07:08:40 UTC, 86.5% complete 2024-04-17 10:10:20,016 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb19.index from /ocean/dlatorne/oxygen-feb19.tar 2024-04-17 10:10:20,119 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-feb19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-04-17 10:14:16,411 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17598600 = 2024-04-19 11:59:20 UTC, 100.0% complete 2024-04-17 10:19:16,418 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2024-04-17 10:19:16,422 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-17 10:19:16,651 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-17'], host='arbutus.cloud-nowcast') 2024-04-17 10:19:16,658 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-17'], host='arbutus.cloud-nowcast') 2024-04-17 10:19:16,662 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:19:16,669 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:19:16,674 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:19:16,684 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:19:16,699 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:19:18,054 INFO [upload_forcing] running in process 2013544 2024-04-17 10:19:18,056 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:18,057 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-04-17 10:19:18,058 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:18,059 INFO [download_results] running in process 2013546 2024-04-17 10:19:18,061 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:18,062 INFO [download_results] publishing log messages to tcp://*:5567 2024-04-17 10:19:18,063 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:18,064 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/17apr24 to /results/SalishSea/forecast.202111 2024-04-17 10:19:18,067 INFO [upload_forcing] running in process 2013543 2024-04-17 10:19:18,067 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:18,068 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-04-17 10:19:18,069 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:18,071 INFO [upload_forcing] running in process 2013542 2024-04-17 10:19:18,072 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:18,074 INFO [upload_forcing] publishing log messages to tcp://*:5569 2024-04-17 10:19:18,075 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:18,148 INFO [upload_forcing] running in process 2013541 2024-04-17 10:19:18,150 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:18,152 INFO [upload_forcing] publishing log messages to tcp://*:5570 2024-04-17 10:19:18,153 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:18,844 INFO [upload_forcing] turbidity 2024-04-17 forcing files upload to optimum-hindcast completed 2024-04-17 10:19:18,848 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-17 10:19:19,289 INFO [upload_forcing] turbidity 2024-04-17 forcing files upload to orcinus-nowcast-agrif completed 2024-04-17 10:19:19,294 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-17 10:19:19,304 INFO [upload_forcing] turbidity 2024-04-17 forcing files upload to arbutus.cloud-nowcast completed 2024-04-17 10:19:19,314 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:19:19,321 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-17 10:19:19,336 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:19:20,530 INFO [make_forcing_links] running in process 2013577 2024-04-17 10:19:20,531 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:20,532 INFO [make_forcing_links] publishing log messages to tcp://*:5566 2024-04-17 10:19:20,534 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:20,554 INFO [make_forcing_links] running in process 2013579 2024-04-17 10:19:20,556 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:20,557 INFO [make_forcing_links] publishing log messages to tcp://*:5569 2024-04-17 10:19:20,559 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:21,477 INFO [make_forcing_links] nowcast-agrif 2024-04-17 forcing file links on orcinus-nowcast-agrif created 2024-04-17 10:19:21,500 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-17 10:19:21,501 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:19:21,912 INFO [make_forcing_links] nowcast-green 2024-04-17 forcing file links on arbutus.cloud-nowcast created 2024-04-17 10:19:21,916 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-17 10:19:21,935 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-17'], host='arbutus.cloud-nowcast') 2024-04-17 10:19:22,117 INFO [upload_forcing] turbidity 2024-04-17 forcing files upload to robot.graham completed 2024-04-17 10:19:22,122 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-17 10:19:22,914 INFO [run_NEMO_agrif] running in process 2013593 2024-04-17 10:19:22,923 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:22,924 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5566 2024-04-17 10:19:22,925 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:25,372 INFO [run_NEMO] running in process 29677 2024-04-17 10:19:25,373 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:25,374 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-04-17 10:19:25,458 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:26,431 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/17apr24nowcast-green_2024-04-17T101925.419354-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-17 10:19:26,618 INFO [run_NEMO] nowcast-green NEMO run for 2024-04-17 on arbutus.cloud-nowcast started 2024-04-17 10:19:26,640 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-17 10:19:26,666 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2024-04-17 10:19:28,913 INFO [watch_NEMO] running in process 29942 2024-04-17 10:19:28,916 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:28,917 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-17 10:19:28,918 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:29,040 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-17 10:19:45,763 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-04-17 10:19:52,439 INFO [run_NEMO_agrif] job id for 17apr24nowcast-agrif: 11091786.orca2.ibb 2024-04-17 10:19:52,440 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-04-17 queued on orcinus-nowcast-agrif 2024-04-17 10:19:52,451 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker 2024-04-17 10:19:52,473 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11091786.orca2.ibb'], host='localhost') 2024-04-17 10:19:53,720 INFO [watch_NEMO_agrif] running in process 2013613 2024-04-17 10:19:53,721 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:19:53,722 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5566 2024-04-17 10:19:53,724 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:19:54,507 INFO [watch_NEMO_agrif] watching 17apr24nowcast-agrif job 11091786 on orcinus-nowcast-agrif 2024-04-17 10:19:54,692 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif job 11091786 is queued on orcinus-nowcast-agrif 2024-04-17 10:20:55,439 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2024-04-17 10:24:29,041 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-17 10:25:55,814 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7614201 = 2024-04-17 02:13:20 UTC, 9.3% complete 2024-04-17 10:29:29,033 CRITICAL [watch_NEMO] arbutus.cloud-nowcast nowcast-green/17apr24 run failed; no time.step file 2024-04-17 10:29:29,340 CRITICAL [watch_NEMO] arbutus.cloud-nowcast nowcast-green/17apr24 run failed; no physics restart file: /nemoShare/MEOPAR/SalishSea/nowcast-green/17apr24/SalishSea_17595360_restart.nc 2024-04-17 10:29:29,364 CRITICAL [watch_NEMO] arbutus.cloud-nowcast nowcast-green/17apr24 run failed; no tracers restart file: /nemoShare/MEOPAR/SalishSea/nowcast-green/17apr24/SalishSea_17595360_restart_trc.nc 2024-04-17 10:29:29,392 CRITICAL [watch_NEMO] arbutus.cloud-nowcast nowcast-green/17apr24 run failed; 1 or more E R R O R in: /nemoShare/MEOPAR/SalishSea/nowcast-green/17apr24/ocean.output 2024-04-17 10:29:29,418 CRITICAL [watch_NEMO] arbutus.cloud-nowcast nowcast-green/17apr24 run failed; no solver.stat file 2024-04-17 10:29:29,442 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2024-04-17 10:29:29,445 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-17 10:29:29,569 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:29:31,099 INFO [download_results] running in process 2013653 2024-04-17 10:29:31,100 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:29:31,102 INFO [download_results] publishing log messages to tcp://*:5568 2024-04-17 10:29:31,103 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:29:31,104 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/17apr24 to /results2/SalishSea/nowcast-green.202111 2024-04-17 10:29:34,537 INFO [download_results] nowcast-green 2024-04-17 results files from arbutus.cloud-nowcast downloaded 2024-04-17 10:29:34,542 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 10:29:34,561 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:29:34,565 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2024-04-17 10:29:34,569 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:29:34,574 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:29:34,578 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:29:35,972 INFO [ping_erddap] running in process 2013661 2024-04-17 10:29:35,973 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:29:35,975 INFO [ping_erddap] publishing log messages to tcp://*:5568 2024-04-17 10:29:35,976 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:29:35,981 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2024-04-17 10:29:35,983 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-17 10:29:38,383 INFO [make_averaged_dataset] running in process 2013664 2024-04-17 10:29:38,384 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:29:38,385 INFO [make_averaged_dataset] publishing log messages to tcp://*:5568 2024-04-17 10:29:38,388 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:29:38,390 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Apr-2024 biology 2024-04-17 10:29:38,391 INFO [make_averaged_dataset] running in process 2013662 2024-04-17 10:29:38,393 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:29:38,394 INFO [make_averaged_dataset] publishing log messages to tcp://*:5569 2024-04-17 10:29:38,396 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:29:38,398 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Apr-2024 physics 2024-04-17 10:29:38,399 INFO [make_averaged_dataset] running in process 2013663 2024-04-17 10:29:38,401 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:29:38,402 INFO [make_averaged_dataset] publishing log messages to tcp://*:5570 2024-04-17 10:29:38,403 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:29:38,404 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Apr-2024 chemistry 2024-04-17 10:29:41,448 INFO [make_plots] running in process 2013659 2024-04-17 10:29:41,449 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:29:41,451 INFO [make_plots] publishing log messages to tcp://*:5571 2024-04-17 10:29:41,452 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:29:42,162 INFO [make_plots] preparing render list for 2024-04-17 NEMO nowcast-green research figures 2024-04-17 10:29:42,169 CRITICAL [make_plots] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 201, in make_plots fig_functions = _prep_nowcast_green_research_fig_functions( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 448, in _prep_nowcast_green_research_fig_functions grid_T_hr = _results_dataset("1h", "grid_T", results_dir) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 346, in _results_dataset return nc.Dataset(filepaths[0]) ~~~~~~~~~^^^ IndexError: list index out of range 2024-04-17 10:30:09,983 CRITICAL [make_averaged_dataset] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 210, in _acquire_with_cache_info file = self._cache[self._key] ~~~~~~~~~~~^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__ value = self._cache[key] ~~~~~~~~~~~^^^^^ KeyError: [, ('/results2/SalishSea/nowcast-green.202111/17apr24/SalishSea_1h_20240417_20240417_grid_T.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), 'ee472c1d-25e4-47c2-b7dc-3734bad64cce'] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_averaged_dataset.py", line 183, in make_averaged_dataset nc_path = _extract_netcdf(reshapr_config, reshapr_config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 289, in wrapped_f return self(f, *args, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 379, in __call__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 382, in __call__ result = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_averaged_dataset.py", line 206, in _extract_netcdf nc_path = reshapr.api.v1.extract.extract_netcdf(reshapr_config, reshapr_config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/api/v1/extract.py", line 49, in extract_netcdf return extract.api_extract_netcdf(config, config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/core/extract.py", line 62, in api_extract_netcdf with open_dataset(ds_paths, chunk_size, extract_config) as ds: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/core/extract.py", line 482, in open_dataset with xarray.open_dataset(ds_path, chunks=chunk_size) as ds: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/api.py", line 526, in open_dataset backend_ds = backend.open_dataset( ^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 577, in open_dataset store = NetCDF4DataStore.open( ^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 382, in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 329, in __init__ self.format = self.ds.data_model ^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 391, in ds return self._acquire() ^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 385, in _acquire with self._manager.acquire_context(needs_lock) as root: File "/SalishSeaCast/nowcast-env/lib/python3.11/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 198, in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 216, in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "src/netCDF4/_netCDF4.pyx", line 2449, in netCDF4._netCDF4.Dataset.__init__ File "src/netCDF4/_netCDF4.pyx", line 2012, in netCDF4._netCDF4._ensure_nc_success FileNotFoundError: [Errno 2] No such file or directory: '/results2/SalishSea/nowcast-green.202111/17apr24/SalishSea_1h_20240417_20240417_grid_T.nc' 2024-04-17 10:30:10,490 CRITICAL [make_averaged_dataset] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 210, in _acquire_with_cache_info file = self._cache[self._key] ~~~~~~~~~~~^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__ value = self._cache[key] ~~~~~~~~~~~^^^^^ KeyError: [, ('/results2/SalishSea/nowcast-green.202111/17apr24/SalishSea_1h_20240417_20240417_chem_T.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '9ef3149d-9c34-40f1-ac6a-1fd2134e44ca'] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_averaged_dataset.py", line 183, in make_averaged_dataset nc_path = _extract_netcdf(reshapr_config, reshapr_config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 289, in wrapped_f return self(f, *args, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 379, in __call__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 382, in __call__ result = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_averaged_dataset.py", line 206, in _extract_netcdf nc_path = reshapr.api.v1.extract.extract_netcdf(reshapr_config, reshapr_config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/api/v1/extract.py", line 49, in extract_netcdf return extract.api_extract_netcdf(config, config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/core/extract.py", line 62, in api_extract_netcdf with open_dataset(ds_paths, chunk_size, extract_config) as ds: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/core/extract.py", line 482, in open_dataset with xarray.open_dataset(ds_path, chunks=chunk_size) as ds: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/api.py", line 526, in open_dataset backend_ds = backend.open_dataset( ^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 577, in open_dataset store = NetCDF4DataStore.open( ^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 382, in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 329, in __init__ self.format = self.ds.data_model ^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 391, in ds return self._acquire() ^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 385, in _acquire with self._manager.acquire_context(needs_lock) as root: File "/SalishSeaCast/nowcast-env/lib/python3.11/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 198, in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 216, in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "src/netCDF4/_netCDF4.pyx", line 2449, in netCDF4._netCDF4.Dataset.__init__ File "src/netCDF4/_netCDF4.pyx", line 2012, in netCDF4._netCDF4._ensure_nc_success FileNotFoundError: [Errno 2] No such file or directory: '/results2/SalishSea/nowcast-green.202111/17apr24/SalishSea_1h_20240417_20240417_chem_T.nc' 2024-04-17 10:30:10,817 CRITICAL [make_averaged_dataset] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 210, in _acquire_with_cache_info file = self._cache[self._key] ~~~~~~~~~~~^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__ value = self._cache[key] ~~~~~~~~~~~^^^^^ KeyError: [, ('/results2/SalishSea/nowcast-green.202111/17apr24/SalishSea_1h_20240417_20240417_biol_T.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '3f2ef279-c7b6-409b-bc31-7ea277f79281'] During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_averaged_dataset.py", line 183, in make_averaged_dataset nc_path = _extract_netcdf(reshapr_config, reshapr_config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 289, in wrapped_f return self(f, *args, **kw) ^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 379, in __call__ do = self.iter(retry_state=retry_state) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 325, in iter raise retry_exc.reraise() ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 158, in reraise raise self.last_attempt.result() ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/concurrent/futures/_base.py", line 449, in result return self.__get_result() ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result raise self._exception File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/tenacity/__init__.py", line 382, in __call__ result = fn(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_averaged_dataset.py", line 206, in _extract_netcdf nc_path = reshapr.api.v1.extract.extract_netcdf(reshapr_config, reshapr_config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/api/v1/extract.py", line 49, in extract_netcdf return extract.api_extract_netcdf(config, config_yaml) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/core/extract.py", line 62, in api_extract_netcdf with open_dataset(ds_paths, chunk_size, extract_config) as ds: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/Reshapr/reshapr/core/extract.py", line 482, in open_dataset with xarray.open_dataset(ds_path, chunks=chunk_size) as ds: ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/api.py", line 526, in open_dataset backend_ds = backend.open_dataset( ^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 577, in open_dataset store = NetCDF4DataStore.open( ^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 382, in open return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 329, in __init__ self.format = self.ds.data_model ^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 391, in ds return self._acquire() ^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/netCDF4_.py", line 385, in _acquire with self._manager.acquire_context(needs_lock) as root: File "/SalishSeaCast/nowcast-env/lib/python3.11/contextlib.py", line 137, in __enter__ return next(self.gen) ^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 198, in acquire_context file, cached = self._acquire_with_cache_info(needs_lock) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/backends/file_manager.py", line 216, in _acquire_with_cache_info file = self._opener(*self._args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "src/netCDF4/_netCDF4.pyx", line 2449, in netCDF4._netCDF4.Dataset.__init__ File "src/netCDF4/_netCDF4.pyx", line 2012, in netCDF4._netCDF4._ensure_nc_success FileNotFoundError: [Errno 2] No such file or directory: '/results2/SalishSea/nowcast-green.202111/17apr24/SalishSea_1h_20240417_20240417_biol_T.nc' 2024-04-17 10:30:56,371 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7614407 = 2024-04-17 04:30:40 UTC, 18.8% complete 2024-04-17 10:35:56,880 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7614613 = 2024-04-17 06:48:00 UTC, 28.3% complete 2024-04-17 10:40:57,364 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7614819 = 2024-04-17 09:05:20 UTC, 37.9% complete 2024-04-17 10:45:57,824 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7615026 = 2024-04-17 11:23:20 UTC, 47.5% complete 2024-04-17 10:48:45,494 INFO [download_results] forecast 2024-04-17 results files from arbutus.cloud-nowcast downloaded 2024-04-17 10:48:45,499 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 10:48:45,523 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:48:48,147 INFO [make_CHS_currents_file] running in process 2014042 2024-04-17 10:48:48,148 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:48:48,149 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5567 2024-04-17 10:48:48,150 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:49:33,166 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-17 for forecast 2024-04-17 10:49:33,168 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-04-17 10:49:33,182 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:49:34,253 INFO [update_forecast_datasets] running in process 2014070 2024-04-17 10:49:34,254 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:49:34,256 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567 2024-04-17 10:49:34,257 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:49:34,258 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/17apr24 run 2024-04-17 10:49:34,419 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/17apr24 run: /results/SalishSea/rolling-forecasts/nemo 2024-04-17 10:49:34,420 INFO [update_forecast_datasets] nemo 2024-04-17 forecast rolling forecast datasets updated 2024-04-17 10:49:34,422 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-17 10:49:34,428 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-04-17 10:49:34,433 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:49:34,438 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 10:49:35,424 INFO [ping_erddap] running in process 2014075 2024-04-17 10:49:35,425 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:49:35,426 INFO [ping_erddap] publishing log messages to tcp://*:5567 2024-04-17 10:49:35,427 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:49:35,438 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-04-17 10:49:35,439 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-17 10:49:37,235 INFO [make_surface_current_tiles] running in process 2014078 2024-04-17 10:49:37,236 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:49:37,237 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5567 2024-04-17 10:49:37,238 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:49:37,240 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-17 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/17apr24 2024-04-17 10:49:39,236 INFO [make_plots] running in process 2014077 2024-04-17 10:49:39,237 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:49:39,238 INFO [make_plots] publishing log messages to tcp://*:5568 2024-04-17 10:49:39,239 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:49:39,871 INFO [make_plots] preparing render list for 2024-04-17 NEMO forecast publish figures 2024-04-17 10:49:39,993 INFO [make_plots] added Website_thumbnail figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,017 INFO [make_plots] added Threshold_website figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,045 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,050 INFO [make_plots] added Boundary Bay figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,051 INFO [make_plots] added Campbell River figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,052 INFO [make_plots] added Cherry Point figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,053 INFO [make_plots] added Friday Harbor figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,054 INFO [make_plots] added Halfmoon Bay figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,055 INFO [make_plots] added Nanaimo figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,056 INFO [make_plots] added Neah Bay figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,057 INFO [make_plots] added New Westminster figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,058 INFO [make_plots] added Patricia Bay figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,059 INFO [make_plots] added Point Atkinson figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,060 INFO [make_plots] added Port Renfrew figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,061 INFO [make_plots] added Sand Heads figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,062 INFO [make_plots] added Sandy Cove figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,063 INFO [make_plots] added Squamish figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,064 INFO [make_plots] added Victoria figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,064 INFO [make_plots] added Woodwards Landing figure to 2024-04-17 NEMO forecast publish render list 2024-04-17 10:49:40,065 INFO [make_plots] starting to render nemo forecast publish 17apr24 figures 2024-04-17 10:50:58,203 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7615232 = 2024-04-17 13:40:40 UTC, 57.0% complete 2024-04-17 10:54:15,545 INFO [make_forcing_links] running in process 2014920 2024-04-17 10:54:15,545 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:54:15,555 INFO [make_forcing_links] publishing log messages to tcp://*:5569 2024-04-17 10:54:15,556 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:54:17,024 INFO [make_forcing_links] nowcast-green 2024-04-17 forcing file links on arbutus.cloud-nowcast created 2024-04-17 10:54:17,028 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-17 10:54:17,077 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-17'], host='arbutus.cloud-nowcast') 2024-04-17 10:54:19,536 INFO [run_NEMO] running in process 32686 2024-04-17 10:54:19,538 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:54:19,539 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-04-17 10:54:19,540 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:54:20,526 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/17apr24nowcast-green_2024-04-17T105419.590583-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-17 10:54:20,548 INFO [run_NEMO] nowcast-green NEMO run for 2024-04-17 on arbutus.cloud-nowcast started 2024-04-17 10:54:20,551 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-17 10:54:20,599 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2024-04-17 10:54:22,591 INFO [watch_NEMO] running in process 474 2024-04-17 10:54:22,592 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 10:54:22,594 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-17 10:54:22,595 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-17 10:54:22,663 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-17 10:55:58,675 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7615439 = 2024-04-17 15:58:40 UTC, 66.6% complete 2024-04-17 10:59:22,666 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17593321 = 2024-04-17 01:20:00 UTC, 5.6% complete 2024-04-17 11:00:59,045 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7615642 = 2024-04-17 18:14:00 UTC, 76.0% complete 2024-04-17 11:04:22,668 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17593497 = 2024-04-17 03:17:20 UTC, 13.7% complete 2024-04-17 11:04:23,364 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/17apr24 2024-04-17 11:05:44,047 INFO [archive_tarball] hindcast *feb19 results files archived to robot.graham 2024-04-17 11:05:44,050 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-04-17 11:05:59,451 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7615848 = 2024-04-17 20:31:20 UTC, 85.5% complete 2024-04-17 11:08:40,469 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/17apr24 2024-04-17 11:08:40,478 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-17 forecast into /results/nowcast-sys/figures/surface_currents/forecast/17apr24 2024-04-17 11:08:40,479 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-17 forecast completed 2024-04-17 11:08:40,801 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-04-17 11:08:54,668 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-04-17 11:08:54,670 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-04-17 11:09:22,671 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17593674 = 2024-04-17 05:15:20 UTC, 21.9% complete 2024-04-17 11:09:31,993 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:10:59,934 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7616055 = 2024-04-17 22:49:20 UTC, 95.1% complete 2024-04-17 11:11:27,524 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:11:27,925 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:12:05,228 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:13:22,044 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:14:00,557 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:14:22,671 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17593851 = 2024-04-17 07:13:20 UTC, 30.1% complete 2024-04-17 11:14:39,865 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:15:17,345 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:15:55,844 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:16:00,395 INFO [watch_NEMO_agrif] 17apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7616160 = 2024-04-17 23:59:20 UTC, 100.0% complete 2024-04-17 11:16:34,859 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:17:14,465 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:17:14,909 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:17:52,898 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:18:31,696 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-12 00:00:00Z to 2024-04-19 11:55:00Z 2024-04-17 11:19:11,525 INFO [make_plots] finished rendering nemo forecast publish 17apr24 figures 2024-04-17 11:19:11,527 INFO [make_plots] nemo publish plots for 2024-04-17 forecast completed 2024-04-17 11:19:11,542 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-17 11:19:11,723 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 11:19:15,906 INFO [make_feeds] running in process 2016246 2024-04-17 11:19:15,907 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 11:19:15,909 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-04-17 11:19:15,910 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-04-17 11:19:22,672 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17594028 = 2024-04-17 09:11:20 UTC, 38.3% complete 2024-04-17 11:19:46,327 INFO [make_feeds] ATOM feeds for 2024-04-17 forecast run completed 2024-04-17 11:19:46,356 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-04-17 11:21:00,648 INFO [watch_NEMO_agrif] NEMO AGRIF run 11091786.orca2.ibb on orcinus-nowcast-agrif completed 2024-04-17 11:21:00,652 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker 2024-04-17 11:21:01,054 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 11:21:02,171 INFO [download_results] running in process 2016273 2024-04-17 11:21:02,172 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 11:21:02,173 INFO [download_results] publishing log messages to tcp://*:5563 2024-04-17 11:21:02,174 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 11:21:02,175 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/17apr24 to /results/SalishSea/nowcast-agrif.201702 2024-04-17 11:21:31,444 INFO [download_results] nowcast-agrif 2024-04-17 results files from orcinus-nowcast-agrif downloaded 2024-04-17 11:21:31,457 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 11:21:31,658 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 11:21:36,140 INFO [make_plots] running in process 2016280 2024-04-17 11:21:36,142 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 11:21:36,143 INFO [make_plots] publishing log messages to tcp://*:5563 2024-04-17 11:21:36,144 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-17 11:21:36,904 INFO [make_plots] preparing render list for 2024-04-17 NEMO nowcast-agrif research figures 2024-04-17 11:21:36,904 INFO [make_plots] added baynes_sound_surface figure to 2024-04-17 NEMO nowcast-agrif research render list 2024-04-17 11:21:36,905 INFO [make_plots] starting to render nemo nowcast-agrif research 17apr24 figures 2024-04-17 11:21:46,474 INFO [make_plots] finished rendering nemo nowcast-agrif research 17apr24 figures 2024-04-17 11:21:46,477 INFO [make_plots] nemo research plots for 2024-04-17 nowcast-agrif completed 2024-04-17 11:21:46,481 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-17 11:24:22,676 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17594205 = 2024-04-17 11:09:20 UTC, 46.5% complete 2024-04-17 11:29:22,672 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17594381 = 2024-04-17 13:06:40 UTC, 54.7% complete 2024-04-17 11:32:25,775 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 11:32:26,277 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-01'], host='localhost') 2024-04-17 11:32:26,282 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 11:32:26,288 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 11:32:27,439 INFO [download_results] running in process 2016343 2024-04-17 11:32:27,440 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 11:32:27,441 INFO [download_results] publishing log messages to tcp://*:5563 2024-04-17 11:32:27,442 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 11:32:27,443 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01mar19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 11:32:31,455 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 11:33:27,040 INFO [download_results] hindcast 2019-03-01 results files from optimum-hindcast downloaded 2024-04-17 11:33:27,052 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 11:33:27,229 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-01'], host='localhost') 2024-04-17 11:33:28,298 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 11:34:22,681 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17594557 = 2024-04-17 15:04:00 UTC, 62.8% complete 2024-04-17 11:39:22,676 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17594733 = 2024-04-17 17:01:20 UTC, 71.0% complete 2024-04-17 11:44:22,674 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17594909 = 2024-04-17 18:58:40 UTC, 79.1% complete 2024-04-17 11:49:22,675 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17595084 = 2024-04-17 20:55:20 UTC, 87.2% complete 2024-04-17 11:54:22,682 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17595259 = 2024-04-17 22:52:00 UTC, 95.3% complete 2024-04-17 11:59:22,676 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17595360 = 2024-04-17 23:59:20 UTC, 100.0% complete 2024-04-17 12:04:22,684 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2024-04-17 12:04:22,687 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-17 12:04:23,086 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 12:04:24,306 INFO [download_results] running in process 2016495 2024-04-17 12:04:24,308 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 12:04:24,309 INFO [download_results] publishing log messages to tcp://*:5563 2024-04-17 12:04:24,310 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 12:04:24,311 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/17apr24 to /results2/SalishSea/nowcast-green.202111 2024-04-17 12:38:22,775 INFO [download_results] nowcast-green 2024-04-17 results files from arbutus.cloud-nowcast downloaded 2024-04-17 12:38:22,782 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 12:38:22,969 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 12:38:22,973 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2024-04-17 12:38:22,977 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 12:38:22,982 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 12:38:22,986 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 12:38:24,285 INFO [ping_erddap] running in process 2016715 2024-04-17 12:38:24,286 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 12:38:24,287 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-04-17 12:38:24,288 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-17 12:38:24,294 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2024-04-17 12:38:24,296 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-17 12:38:26,910 INFO [make_averaged_dataset] running in process 2016718 2024-04-17 12:38:26,912 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 12:38:26,913 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563 2024-04-17 12:38:26,914 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-17 12:38:26,915 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Apr-2024 biology 2024-04-17 12:38:26,984 INFO [make_averaged_dataset] running in process 2016717 2024-04-17 12:38:27,001 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 12:38:27,002 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566 2024-04-17 12:38:27,005 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-17 12:38:27,007 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Apr-2024 chemistry 2024-04-17 12:38:27,009 INFO [make_averaged_dataset] running in process 2016716 2024-04-17 12:38:27,010 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 12:38:27,011 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567 2024-04-17 12:38:27,013 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-17 12:38:27,013 INFO [make_averaged_dataset] creating day-averaged dataset for 17-Apr-2024 physics 2024-04-17 12:38:30,181 INFO [make_plots] running in process 2016713 2024-04-17 12:38:30,183 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 12:38:30,184 INFO [make_plots] publishing log messages to tcp://*:5568 2024-04-17 12:38:30,190 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-17 12:38:31,037 INFO [make_plots] preparing render list for 2024-04-17 NEMO nowcast-green research figures 2024-04-17 12:38:32,957 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:33,712 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:34,441 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:35,217 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:36,002 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:36,746 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:37,492 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:38,337 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:39,201 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:40,026 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:40,835 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:41,554 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:41,844 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:41,845 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:41,846 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:41,847 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-04-17 NEMO nowcast-green research render list 2024-04-17 12:38:41,848 INFO [make_plots] starting to render nemo nowcast-green research 17apr24 figures 2024-04-17 12:39:30,695 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-04-17 12:39:30,927 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-17'], host='arbutus.cloud-nowcast') 2024-04-17 12:39:33,008 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-17 12:39:33,176 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-04-17 12:39:35,135 INFO [watch_ww3] running in process 8686 2024-04-17 12:39:35,136 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 12:39:35,137 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-04-17 12:39:35,138 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-17 12:39:35,177 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-17 12:40:05,907 INFO [make_averaged_dataset] day-averaged dataset for 17-Apr-2024 chemistry created 2024-04-17 12:40:05,909 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-04-17 12:40:15,566 INFO [make_averaged_dataset] day-averaged dataset for 17-Apr-2024 physics created 2024-04-17 12:40:15,570 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-04-17 12:40:53,210 INFO [make_averaged_dataset] day-averaged dataset for 17-Apr-2024 biology created 2024-04-17 12:40:53,213 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-04-17 12:44:35,169 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete 2024-04-17 12:49:35,170 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete 2024-04-17 12:54:35,170 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 300, 69.4% complete 2024-04-17 12:57:36,248 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 12:57:36,726 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-06'], host='localhost') 2024-04-17 12:57:36,733 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 12:57:36,737 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 12:57:38,420 INFO [download_results] running in process 2016981 2024-04-17 12:57:38,420 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 12:57:38,422 INFO [download_results] publishing log messages to tcp://*:5563 2024-04-17 12:57:38,423 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 12:57:38,424 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06mar19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 12:57:42,659 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 12:59:35,197 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 399, 92.4% complete 2024-04-17 12:59:35,199 INFO [download_results] hindcast 2019-03-06 results files from optimum-hindcast downloaded 2024-04-17 12:59:35,212 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 12:59:35,487 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-06'], host='localhost') 2024-04-17 12:59:37,002 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 13:04:35,172 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-17 13:04:35,192 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-17 13:04:35,611 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 13:04:35,618 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-17'], host='arbutus.cloud-nowcast') 2024-04-17 13:04:37,184 INFO [download_wwatch3_results] running in process 2017091 2024-04-17 13:04:37,185 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 13:04:37,186 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563 2024-04-17 13:04:37,187 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 13:04:37,604 INFO [run_ww3] running in process 10532 2024-04-17 13:04:37,606 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 13:04:37,607 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-04-17 13:04:37,608 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-17 13:04:37,612 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-17T130437.614863-0700 2024-04-17 13:04:37,613 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-17T130437.614863-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-04-17 13:04:37,632 INFO [run_ww3] forecast WaveWatch3 run for 2024-04-17 on arbutus.cloud-nowcast started 2024-04-17 13:04:37,668 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-17 13:04:37,929 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-04-17 13:04:39,990 INFO [watch_ww3] running in process 10688 2024-04-17 13:04:40,017 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 13:04:40,035 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-04-17 13:04:40,041 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-17 13:04:40,044 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-17 13:06:37,029 INFO [download_wwatch3_results] nowcast 2024-04-17 results files from arbutus.cloud-nowcast downloaded 2024-04-17 13:06:37,033 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-17 13:09:40,037 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete 2024-04-17 13:14:40,199 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete 2024-04-17 13:19:40,176 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 297, 45.8% complete 2024-04-17 13:24:40,095 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 399, 61.6% complete 2024-04-17 13:29:40,179 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete 2024-04-17 13:34:40,154 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 606, 93.5% complete 2024-04-17 13:39:40,061 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-17 13:39:40,114 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-17 13:39:40,620 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 13:39:41,894 INFO [download_wwatch3_results] running in process 2017208 2024-04-17 13:39:41,905 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 13:39:41,923 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563 2024-04-17 13:39:41,941 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 13:42:18,872 INFO [download_wwatch3_results] forecast 2024-04-17 results files from arbutus.cloud-nowcast downloaded 2024-04-17 13:42:18,875 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-17 13:42:19,204 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 13:42:20,480 INFO [update_forecast_datasets] running in process 2017221 2024-04-17 13:42:20,481 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 13:42:20,482 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-04-17 13:42:20,483 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-17 13:42:20,484 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/17apr24 run 2024-04-17 13:42:20,486 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/17apr24 run 2024-04-17 13:42:20,819 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/17apr24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-04-17 13:42:20,820 INFO [update_forecast_datasets] wwatch3 2024-04-17 forecast rolling forecast datasets updated 2024-04-17 13:42:20,823 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-17 13:42:21,052 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-04-17 13:42:22,271 INFO [ping_erddap] running in process 2017226 2024-04-17 13:42:22,272 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 13:42:22,273 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-04-17 13:42:22,274 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-17 13:42:22,276 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-04-17 13:42:22,278 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-17 13:42:22,464 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-04-17'], host='localhost') 2024-04-17 13:42:31,896 INFO [make_plots] running in process 2017231 2024-04-17 13:42:31,897 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 13:42:31,898 INFO [make_plots] publishing log messages to tcp://*:5563 2024-04-17 13:42:31,899 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-17 13:42:31,900 INFO [make_plots] preparing render list for 2024-04-17 WaveWatch3 forecast publish figures 2024-04-17 13:42:31,901 INFO [make_plots] added Halibut Bank figure to 2024-04-17 WaveWatch3 forecast publish render list 2024-04-17 13:42:31,902 INFO [make_plots] added Sentry Shoal figure to 2024-04-17 WaveWatch3 forecast publish render list 2024-04-17 13:42:31,903 INFO [make_plots] starting to render wwatch3 forecast publish 17apr24 figures 2024-04-17 13:42:32,422 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-04-17 13:42:43,675 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-04-17 13:42:50,641 INFO [make_plots] finished rendering wwatch3 forecast publish 17apr24 figures 2024-04-17 13:42:50,650 INFO [make_plots] wwatch3 publish plots for 2024-04-17 forecast completed 2024-04-17 13:42:50,685 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-17 14:04:55,910 INFO [make_plots] finished rendering nemo nowcast-green research 17apr24 figures 2024-04-17 14:04:55,923 INFO [make_plots] nemo research plots for 2024-04-17 nowcast-green completed 2024-04-17 14:04:55,991 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-17 14:22:47,149 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 14:22:47,569 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-11'], host='localhost') 2024-04-17 14:22:47,575 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 14:22:47,580 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 14:22:49,105 INFO [download_results] running in process 2017750 2024-04-17 14:22:49,107 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 14:22:49,108 INFO [download_results] publishing log messages to tcp://*:5566 2024-04-17 14:22:49,109 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 14:22:49,115 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11mar19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 14:22:53,143 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 14:23:52,522 INFO [download_results] hindcast 2019-03-11 results files from optimum-hindcast downloaded 2024-04-17 14:23:52,536 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 14:23:52,772 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-11'], host='localhost') 2024-04-17 14:23:53,970 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 15:09:35,327 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240417/18/ 2024-04-17 15:09:35,345 INFO [collect_weather] 2024-04-17 2.5km weather forecast 18 collection complete 2024-04-17 15:09:35,347 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-04-17 15:09:35,828 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2024-04-17 15:09:35,833 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2024-04-17 15:09:35,839 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2024-04-17 15:09:35,843 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-04-18'], host='localhost') 2024-04-17 15:09:37,125 INFO [collect_weather] running in process 2017967 2024-04-17 15:09:37,127 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 15:09:37,128 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-04-17 15:09:37,128 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-17 15:09:37,129 INFO [download_weather] running in process 2017966 2024-04-17 15:09:37,130 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 15:09:37,131 INFO [download_weather] publishing log messages to tcp://*:5563 2024-04-17 15:09:37,132 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-17 15:09:37,132 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240417 2024-04-17 15:09:37,236 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2024-04-17 15:09:37,237 INFO [download_weather] running in process 2017964 2024-04-17 15:09:37,238 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 15:09:37,239 INFO [download_weather] publishing log messages to tcp://*:5566 2024-04-17 15:09:37,240 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-17 15:09:37,241 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240417 2024-04-17 15:09:39,751 INFO [crop_gribs] running in process 2017968 2024-04-17 15:09:39,752 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 15:09:39,754 INFO [crop_gribs] publishing log messages to tcp://*:5567 2024-04-17 15:09:39,755 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-04-17 15:09:39,756 INFO [crop_gribs] cropping 2024-04-18 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2024-04-17 15:09:39,763 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240418/00/ 2024-04-17 15:14:04,711 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240417/18/ 2024-04-17 15:14:04,712 INFO [crop_gribs] 2024-04-17 18 GRIBs cropping complete 2024-04-17 15:14:04,715 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-04-17 15:17:33,097 INFO [download_weather] 2024-04-17 1km weather forecast 00 downloads complete 2024-04-17 15:17:33,125 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-04-17 15:17:43,091 INFO [download_weather] 2024-04-17 1km weather forecast 12 downloads complete 2024-04-17 15:17:43,094 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-04-17 15:47:57,886 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 15:47:58,368 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-16'], host='localhost') 2024-04-17 15:47:58,375 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 15:47:58,380 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 15:47:59,936 INFO [download_results] running in process 2018303 2024-04-17 15:47:59,947 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 15:47:59,948 INFO [download_results] publishing log messages to tcp://*:5565 2024-04-17 15:47:59,949 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 15:47:59,950 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16mar19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 15:48:04,053 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 15:50:12,404 INFO [download_results] hindcast 2019-03-16 results files from optimum-hindcast downloaded 2024-04-17 15:50:12,420 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 15:50:12,638 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-16'], host='localhost') 2024-04-17 15:50:13,829 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 17:13:08,768 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 17:13:09,262 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-21'], host='localhost') 2024-04-17 17:13:09,268 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 17:13:09,273 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 17:13:10,743 INFO [download_results] running in process 2019265 2024-04-17 17:13:10,745 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 17:13:10,746 INFO [download_results] publishing log messages to tcp://*:5563 2024-04-17 17:13:10,747 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 17:13:10,748 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21mar19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 17:13:14,770 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 17:14:15,402 INFO [download_results] hindcast 2019-03-21 results files from optimum-hindcast downloaded 2024-04-17 17:14:15,428 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 17:14:15,672 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-21'], host='localhost') 2024-04-17 17:14:16,989 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 18:58:21,350 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 18:58:21,780 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-26'], host='localhost') 2024-04-17 18:58:21,789 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 18:58:21,796 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 18:58:23,765 INFO [download_results] running in process 2020355 2024-04-17 18:58:23,766 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 18:58:23,767 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-17 18:58:23,768 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 18:58:23,769 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26mar19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 18:58:27,857 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 18:59:34,910 INFO [download_results] hindcast 2019-03-26 results files from optimum-hindcast downloaded 2024-04-17 18:59:34,924 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 18:59:35,143 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-26'], host='localhost') 2024-04-17 18:59:36,382 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 18:59:36,600 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-mar', 'robot.graham'], host='localhost') 2024-04-17 18:59:37,643 INFO [archive_tarball] running in process 2020406 2024-04-17 18:59:37,644 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 18:59:37,645 INFO [archive_tarball] publishing log messages to tcp://*:5564 2024-04-17 18:59:37,646 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-04-17 18:59:37,647 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar19.tar from /ocean/sallen/timbucktwo/oxygen/*mar19/ 2024-04-17 19:08:45,002 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar19.index from /ocean/dlatorne/oxygen-mar19.tar 2024-04-17 19:08:45,103 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-mar19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-04-17 19:39:18,142 INFO [archive_tarball] hindcast *mar19 results files archived to robot.graham 2024-04-17 19:39:18,147 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-04-17 20:23:32,412 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 20:23:32,830 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-01'], host='localhost') 2024-04-17 20:23:32,837 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 20:23:32,842 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 20:23:34,608 INFO [download_results] running in process 2020640 2024-04-17 20:23:34,609 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 20:23:34,610 INFO [download_results] publishing log messages to tcp://*:5563 2024-04-17 20:23:34,611 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 20:23:34,612 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01apr19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 20:23:38,926 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 20:24:36,498 INFO [download_results] hindcast 2019-04-01 results files from optimum-hindcast downloaded 2024-04-17 20:24:36,513 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 20:24:36,746 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-01'], host='localhost') 2024-04-17 20:24:37,967 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 21:10:00,843 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240418/00/ 2024-04-17 21:10:00,844 INFO [collect_weather] 2024-04-18 2.5km weather forecast 00 collection complete 2024-04-17 21:10:00,847 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-04-17 21:10:01,284 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2024-04-17 21:10:01,291 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-04-18'], host='localhost') 2024-04-17 21:10:03,079 INFO [collect_weather] running in process 2020918 2024-04-17 21:10:03,081 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 21:10:03,082 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-04-17 21:10:03,083 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-17 21:10:03,096 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2024-04-17 21:10:05,573 INFO [crop_gribs] running in process 2020920 2024-04-17 21:10:05,574 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 21:10:05,576 INFO [crop_gribs] publishing log messages to tcp://*:5563 2024-04-17 21:10:05,577 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-04-17 21:10:05,578 INFO [crop_gribs] cropping 2024-04-18 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2024-04-17 21:10:05,586 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240418/06/ 2024-04-17 21:13:57,479 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240418/00/ 2024-04-17 21:13:57,480 INFO [crop_gribs] 2024-04-18 00 GRIBs cropping complete 2024-04-17 21:13:57,482 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-04-17 21:48:44,314 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 21:48:44,819 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-06'], host='localhost') 2024-04-17 21:48:44,827 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 21:48:44,833 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 21:48:46,877 INFO [download_results] running in process 2021054 2024-04-17 21:48:46,879 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 21:48:46,880 INFO [download_results] publishing log messages to tcp://*:5566 2024-04-17 21:48:46,881 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 21:48:46,882 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06apr19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 21:48:50,928 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 21:49:45,795 INFO [download_results] hindcast 2019-04-06 results files from optimum-hindcast downloaded 2024-04-17 21:49:45,808 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 21:49:46,025 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-06'], host='localhost') 2024-04-17 21:49:47,263 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-17 23:13:55,843 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-17 23:13:56,258 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-11'], host='localhost') 2024-04-17 23:13:56,264 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 23:13:56,269 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-17 23:13:57,751 INFO [download_results] running in process 2021309 2024-04-17 23:13:57,752 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-17 23:13:57,753 INFO [download_results] publishing log messages to tcp://*:5565 2024-04-17 23:13:57,754 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-17 23:13:57,755 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11apr19 to /ocean/sallen/timbucktwo/oxygen 2024-04-17 23:14:01,760 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-17 23:16:42,255 INFO [download_results] hindcast 2019-04-11 results files from optimum-hindcast downloaded 2024-04-17 23:16:42,286 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-17 23:16:42,505 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-11'], host='localhost') 2024-04-17 23:16:43,773 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-18 00:39:06,572 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-18 00:39:07,161 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-16'], host='localhost') 2024-04-18 00:39:07,168 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-18 00:39:07,199 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-18 00:39:08,540 INFO [download_results] running in process 2021715 2024-04-18 00:39:08,541 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 00:39:08,542 INFO [download_results] publishing log messages to tcp://*:5565 2024-04-18 00:39:08,544 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-18 00:39:08,545 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16apr19 to /ocean/sallen/timbucktwo/oxygen 2024-04-18 00:39:12,570 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-18 00:40:12,450 INFO [download_results] hindcast 2019-04-16 results files from optimum-hindcast downloaded 2024-04-18 00:40:12,467 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-18 00:40:12,644 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-16'], host='localhost') 2024-04-18 00:40:13,907 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-18 02:04:17,122 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-18 02:04:17,534 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-21'], host='localhost') 2024-04-18 02:04:17,541 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-18 02:04:17,545 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-18 02:04:18,901 INFO [download_results] running in process 2022079 2024-04-18 02:04:18,903 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 02:04:18,904 INFO [download_results] publishing log messages to tcp://*:5566 2024-04-18 02:04:18,906 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-18 02:04:18,906 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21apr19 to /ocean/sallen/timbucktwo/oxygen 2024-04-18 02:04:23,128 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-18 02:05:15,233 INFO [download_results] hindcast 2019-04-21 results files from optimum-hindcast downloaded 2024-04-18 02:05:15,244 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-18 02:05:15,460 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-21'], host='localhost') 2024-04-18 02:05:16,638 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-18 03:09:27,707 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240418/06/ 2024-04-18 03:09:27,708 INFO [collect_weather] 2024-04-18 2.5km weather forecast 06 collection complete 2024-04-18 03:09:27,711 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-04-18 03:09:28,123 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,128 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,133 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,138 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,146 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,150 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,159 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,162 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,169 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,174 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,179 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,188 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,201 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,213 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-04-17'], host='localhost') 2024-04-18 03:09:28,223 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2024-04-18 03:09:28,239 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2024-04-18 03:09:28,250 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2024-04-18 03:09:28,273 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2024-04-18 03:09:28,323 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2024-04-18 03:09:28,359 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2024-04-18 03:09:30,741 INFO [collect_weather] running in process 2022306 2024-04-18 03:09:30,742 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:30,750 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-04-18 03:09:30,751 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:30,754 INFO [collect_NeahBay_ssh] running in process 2022305 2024-04-18 03:09:30,755 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:30,756 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5565 2024-04-18 03:09:30,757 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:30,758 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-04-18 2024-04-18 03:09:30,774 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2024-04-18 03:09:32,932 INFO [collect_river_data] running in process 2022297 2024-04-18 03:09:32,951 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:32,953 INFO [collect_river_data] publishing log messages to tcp://*:5566 2024-04-18 03:09:32,954 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:32,955 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-04-17 2024-04-18 03:09:32,957 INFO [collect_river_data] running in process 2022291 2024-04-18 03:09:32,959 INFO [collect_river_data] running in process 2022299 2024-04-18 03:09:32,959 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:32,960 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:32,961 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-04-18 03:09:32,963 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-04-18 03:09:32,967 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:32,969 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:32,971 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-04-17 2024-04-18 03:09:32,972 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-04-17 2024-04-18 03:09:32,973 INFO [collect_river_data] running in process 2022290 2024-04-18 03:09:32,973 INFO [collect_river_data] running in process 2022300 2024-04-18 03:09:32,974 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:32,975 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:32,976 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-04-18 03:09:32,979 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-04-18 03:09:32,980 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:32,981 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:32,982 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-04-17 2024-04-18 03:09:32,983 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-04-17 2024-04-18 03:09:32,984 INFO [collect_river_data] running in process 2022294 2024-04-18 03:09:32,987 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:32,988 INFO [collect_river_data] publishing log messages to tcp://*:5571 2024-04-18 03:09:32,989 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:32,989 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-04-17 2024-04-18 03:09:33,016 INFO [collect_river_data] running in process 2022286 2024-04-18 03:09:33,019 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,021 INFO [collect_river_data] publishing log messages to tcp://*:5572 2024-04-18 03:09:33,022 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,023 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-04-17 2024-04-18 03:09:33,048 INFO [collect_NeahBay_ssh] 2024-04-18T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2024-04-18 03:09:33,055 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-04-18 03:09:33,063 INFO [collect_river_data] running in process 2022293 2024-04-18 03:09:33,065 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,067 INFO [collect_river_data] publishing log messages to tcp://*:5573 2024-04-18 03:09:33,068 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,068 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-04-17 2024-04-18 03:09:33,108 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2024-04-18 03:09:33,109 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-04-17 completed 2024-04-18 03:09:33,111 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2024-04-18 03:09:33,111 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-04-17 completed 2024-04-18 03:09:33,113 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/RobertsCreek_flow 2024-04-18 03:09:33,114 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-04-17 completed 2024-04-18 03:09:33,116 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2024-04-18 03:09:33,117 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-04-17 completed 2024-04-18 03:09:33,120 INFO [collect_river_data] running in process 2022296 2024-04-18 03:09:33,125 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,126 INFO [collect_river_data] publishing log messages to tcp://*:5574 2024-04-18 03:09:33,128 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,129 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-04-17 2024-04-18 03:09:33,131 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-04-17 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2024-04-18 03:09:33,133 INFO [collect_river_data] ECCC Fraser river data collection for 2024-04-17 completed 2024-04-18 03:09:33,138 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2024-04-18 03:09:33,139 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-04-17 completed 2024-04-18 03:09:33,157 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-04-17 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2024-04-18 03:09:33,158 INFO [collect_river_data] ECCC Capilano river data collection for 2024-04-17 completed 2024-04-18 03:09:33,232 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2024-04-18 03:09:33,249 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-04-17 completed 2024-04-18 03:09:33,273 INFO [collect_river_data] running in process 2022298 2024-04-18 03:09:33,274 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,275 INFO [collect_river_data] publishing log messages to tcp://*:5575 2024-04-18 03:09:33,275 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,276 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-04-17 2024-04-18 03:09:33,306 INFO [collect_river_data] running in process 2022295 2024-04-18 03:09:33,307 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,309 INFO [collect_river_data] publishing log messages to tcp://*:5576 2024-04-18 03:09:33,310 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,313 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-04-17 2024-04-18 03:09:33,313 INFO [collect_river_data] running in process 2022289 2024-04-18 03:09:33,314 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,315 INFO [collect_river_data] publishing log messages to tcp://*:5577 2024-04-18 03:09:33,316 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,317 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-04-17 2024-04-18 03:09:33,319 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-04-17 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2024-04-18 03:09:33,320 INFO [collect_river_data] ECCC Englishman river data collection for 2024-04-17 completed 2024-04-18 03:09:33,334 INFO [collect_river_data] running in process 2022292 2024-04-18 03:09:33,335 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,336 INFO [collect_river_data] publishing log messages to tcp://*:5578 2024-04-18 03:09:33,337 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,338 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-04-17 2024-04-18 03:09:33,347 INFO [collect_river_data] running in process 2022288 2024-04-18 03:09:33,348 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,349 INFO [collect_river_data] publishing log messages to tcp://*:5579 2024-04-18 03:09:33,349 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,350 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-04-17 2024-04-18 03:09:33,368 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2024-04-18 03:09:33,369 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-04-17 completed 2024-04-18 03:09:33,370 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-04-18'], host='localhost') 2024-04-18 03:09:33,377 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2024-04-18 03:09:33,379 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-04-17 completed 2024-04-18 03:09:33,386 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:33,408 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2024-04-18 03:09:33,409 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-04-17 completed 2024-04-18 03:09:33,423 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2024-04-18 03:09:33,423 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-04-17 completed 2024-04-18 03:09:33,449 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-04-17 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2024-04-18 03:09:33,450 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-04-17 completed 2024-04-18 03:09:33,681 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:33,902 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:33,947 INFO [crop_gribs] running in process 2022307 2024-04-18 03:09:33,948 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:33,949 INFO [crop_gribs] publishing log messages to tcp://*:5565 2024-04-18 03:09:33,950 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:33,951 INFO [crop_gribs] cropping 2024-04-18 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2024-04-18 03:09:33,956 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240418/12/ 2024-04-18 03:09:34,191 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:34,434 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:34,715 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:34,801 INFO [get_onc_ctd] running in process 2022303 2024-04-18 03:09:34,802 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:34,803 INFO [get_onc_ctd] publishing log messages to tcp://*:5566 2024-04-18 03:09:34,804 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:34,805 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-04-17 2024-04-18 03:09:34,814 INFO [get_onc_ferry] running in process 2022304 2024-04-18 03:09:34,815 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:34,816 INFO [get_onc_ferry] publishing log messages to tcp://*:5568 2024-04-18 03:09:34,817 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:34,818 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-04-17 2024-04-18 03:09:34,827 INFO [get_onc_ctd] running in process 2022302 2024-04-18 03:09:34,829 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:34,831 INFO [get_onc_ctd] publishing log messages to tcp://*:5570 2024-04-18 03:09:34,832 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:34,833 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-04-17 2024-04-18 03:09:34,969 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:35,171 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:35,374 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:35,573 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:35,812 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:36,019 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:36,262 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:36,477 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-18 03:09:38,138 INFO [make_ssh_files] running in process 2022684 2024-04-18 03:09:38,140 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:38,141 INFO [make_ssh_files] publishing log messages to tcp://*:5567 2024-04-18 03:09:38,142 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:38,143 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-04-18 NOAA Neah Bay 00Z observation and forecast values 2024-04-18 03:09:43,409 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d13.nc 2024-04-18 03:09:43,614 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d14.nc 2024-04-18 03:09:43,824 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d15.nc 2024-04-18 03:09:44,073 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d16.nc 2024-04-18 03:09:44,287 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc 2024-04-18 03:09:44,526 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d18.nc 2024-04-18 03:09:44,768 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d19.nc 2024-04-18 03:09:44,954 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d20.nc 2024-04-18 03:09:45,151 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d21.nc 2024-04-18 03:09:45,172 CRITICAL [make_ssh_files] /results/forcing/sshNeahBay/obs/ssh_y2024m04d17.nc was not created; using ../fcst/ssh_y2024m04d17.nc instead via symlink 2024-04-18 03:09:46,049 INFO [make_ssh_files] sea surface height boundary file for 2024-04-18 forecast2 run created 2024-04-18 03:09:46,054 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-04-18 03:09:46,277 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-04-18 03:09:46,282 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-04-18 03:09:47,753 INFO [make_runoff_file] running in process 2022752 2024-04-18 03:09:47,755 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:47,756 INFO [make_runoff_file] publishing log messages to tcp://*:5567 2024-04-18 03:09:47,757 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:48,041 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-04-18 03:09:48,274 INFO [make_runoff_file] 2024-04-18 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-04-18 03:09:48,276 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-04-18 03:09:48,774 INFO [make_v202111_runoff_file] running in process 2022751 2024-04-18 03:09:48,776 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:48,777 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5567 2024-04-18 03:09:48,778 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:48,779 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-17 2024-04-18 03:09:49,966 ERROR [make_v202111_runoff_file] no 2024-04-17 discharge obs for Skagit_MountVernon: patching 2024-04-18 03:09:50,442 ERROR [make_v202111_runoff_file] no 2024-04-17 discharge obs for Snohomish_Monroe: patching 2024-04-18 03:09:50,682 ERROR [make_v202111_runoff_file] no 2024-04-17 discharge obs for Nisqually_McKenna: patching 2024-04-18 03:09:50,785 ERROR [make_v202111_runoff_file] no 2024-04-17 discharge obs for Greenwater_Greenwater: patching 2024-04-18 03:09:52,572 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-17: /results/forcing/rivers/R202108Dailies_y2024m04d17.nc 2024-04-18 03:09:52,573 INFO [make_v202111_runoff_file] 2024-04-17 runoff file creation completed 2024-04-18 03:09:52,575 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-04-18 03:09:54,194 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-04-17 2024-04-18 03:09:57,796 INFO [get_onc_ctd] 2024-04-17 ONC SEVIP CTD T&S file created 2024-04-18 03:09:57,831 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2024-04-18 03:09:58,017 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2024-04-18 03:09:59,045 INFO [ping_erddap] running in process 2022793 2024-04-18 03:09:59,046 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:09:59,047 INFO [ping_erddap] publishing log messages to tcp://*:5566 2024-04-18 03:09:59,048 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:09:59,050 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2024-04-18 03:09:59,052 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-18 03:10:10,095 CRITICAL [get_onc_ctd] 2024-04-17 ONC SCVIP CTD T&S file creation failed 2024-04-18 03:10:21,118 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-04-17 2024-04-18 03:10:49,928 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-04-17 2024-04-18 03:11:22,885 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-04-17 2024-04-18 03:11:40,047 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-04-17 2024-04-18 03:11:58,100 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-04-17 2024-04-18 03:12:06,417 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-04-17 2024-04-18 03:12:15,123 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-04-17; substituting empty dataset 2024-04-18 03:12:15,123 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-04-17 2024-04-18 03:12:15,227 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240418/06/ 2024-04-18 03:12:15,228 INFO [crop_gribs] 2024-04-18 06 GRIBs cropping complete 2024-04-18 03:12:15,230 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-04-18 03:12:15,642 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2024-04-18 03:12:18,089 INFO [grib_to_netcdf] running in process 2022806 2024-04-18 03:12:18,090 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:12:18,092 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563 2024-04-18 03:12:18,093 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:12:18,094 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-18 forecast2 run 2024-04-18 03:12:23,815 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-17; substituting empty dataset 2024-04-18 03:12:24,446 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-04-17 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-04-18 03:12:24,467 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-17 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-04-18 03:12:24,891 INFO [get_onc_ferry] 2024-04-17 ONC TWDP ferry data file created 2024-04-18 03:12:24,919 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker 2024-04-18 03:12:25,111 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2024-04-18 03:12:26,140 INFO [ping_erddap] running in process 2022832 2024-04-18 03:12:26,141 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:12:26,143 INFO [ping_erddap] publishing log messages to tcp://*:5566 2024-04-18 03:12:26,144 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:12:26,145 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2024-04-18 03:12:26,147 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-18 03:12:34,279 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d19.nc 2024-04-18 03:12:37,997 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d20.nc 2024-04-18 03:12:38,006 INFO [grib_to_netcdf] 2024-04-18 NEMO-atmos forcing file for forecast2 created 2024-04-18 03:12:38,009 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-04-18 03:12:38,192 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2024-04-18 03:12:38,197 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2024-04-18 03:12:38,199 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2024-04-18 03:12:38,203 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2024-04-18 03:12:39,477 INFO [upload_forcing] running in process 2022851 2024-04-18 03:12:39,480 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:12:39,481 INFO [upload_forcing] publishing log messages to tcp://*:5563 2024-04-18 03:12:39,482 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:12:39,484 INFO [upload_forcing] running in process 2022853 2024-04-18 03:12:39,485 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:12:39,486 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-04-18 03:12:39,487 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:12:39,500 INFO [upload_forcing] running in process 2022854 2024-04-18 03:12:39,502 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:12:39,503 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-04-18 03:12:39,504 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:12:39,505 INFO [upload_forcing] running in process 2022855 2024-04-18 03:12:39,506 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:12:39,507 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-04-18 03:12:39,508 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:12:41,030 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m04d17.nc 2024-04-18 03:12:41,109 INFO [upload_forcing] forecast2 2024-04-18 forcing files upload to optimum-hindcast completed 2024-04-18 03:12:41,111 INFO [upload_forcing] forecast2 2024-04-18 forcing files upload to orcinus-nowcast-agrif completed 2024-04-18 03:12:41,113 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-18 03:12:41,335 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-18 03:12:41,855 INFO [upload_forcing] forecast2 2024-04-18 forcing files upload to arbutus.cloud-nowcast completed 2024-04-18 03:12:41,870 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-18 03:12:42,111 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-18'], host='localhost') 2024-04-18 03:12:43,393 INFO [make_forcing_links] running in process 2022882 2024-04-18 03:12:43,402 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:12:43,420 INFO [make_forcing_links] publishing log messages to tcp://*:5563 2024-04-18 03:12:43,438 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:12:44,454 INFO [make_forcing_links] forecast2 2024-04-18 forcing file links on arbutus.cloud-nowcast created 2024-04-18 03:12:44,458 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-18 03:12:44,684 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-18'], host='arbutus.cloud-nowcast') 2024-04-18 03:12:46,786 INFO [upload_forcing] forecast2 2024-04-18 forcing files upload to robot.graham completed 2024-04-18 03:12:46,796 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-18 03:12:47,972 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/17apr24forecast2_2024-04-18T031247.097143-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-18 03:12:47,992 INFO [run_NEMO] forecast2 NEMO run for 2024-04-18 on arbutus.cloud-nowcast started 2024-04-18 03:12:47,997 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-18 03:12:48,215 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-04-18 03:12:50,067 INFO [watch_NEMO] running in process 9857 2024-04-18 03:12:50,086 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:12:50,104 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-18 03:12:50,122 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:12:50,138 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-18 03:17:50,141 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17597949 = 2024-04-19 04:45:20 UTC, 15.9% complete 2024-04-18 03:22:50,143 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17598430 = 2024-04-19 10:06:00 UTC, 33.7% complete 2024-04-18 03:27:50,145 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17598908 = 2024-04-19 15:24:40 UTC, 51.4% complete 2024-04-18 03:32:50,147 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17599413 = 2024-04-19 21:01:20 UTC, 70.1% complete 2024-04-18 03:34:27,959 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-18 03:34:28,387 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-26'], host='localhost') 2024-04-18 03:34:28,393 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-18 03:34:28,399 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-18 03:34:29,656 INFO [download_results] running in process 2022932 2024-04-18 03:34:29,657 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:34:29,659 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-18 03:34:29,660 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:34:29,661 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26apr19 to /ocean/sallen/timbucktwo/oxygen 2024-04-18 03:34:33,754 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-18 03:35:22,551 INFO [download_results] hindcast 2019-04-26 results files from optimum-hindcast downloaded 2024-04-18 03:35:22,566 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-18 03:35:22,782 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-26'], host='localhost') 2024-04-18 03:35:23,770 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-18 03:35:23,984 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-apr', 'robot.graham'], host='localhost') 2024-04-18 03:35:24,800 INFO [archive_tarball] running in process 2022977 2024-04-18 03:35:24,801 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:35:24,802 INFO [archive_tarball] publishing log messages to tcp://*:5563 2024-04-18 03:35:24,804 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:35:24,805 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-apr19.tar from /ocean/sallen/timbucktwo/oxygen/*apr19/ 2024-04-18 03:37:50,148 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17599851 = 2024-04-20 01:53:20 UTC, 86.3% complete 2024-04-18 03:42:17,429 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-apr19.index from /ocean/dlatorne/oxygen-apr19.tar 2024-04-18 03:42:17,547 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-apr19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-04-18 03:42:50,149 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17600220 = 2024-04-20 05:59:20 UTC, 100.0% complete 2024-04-18 03:47:50,152 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2024-04-18 03:47:50,156 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-18 03:47:50,619 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-18'], host='arbutus.cloud-nowcast') 2024-04-18 03:47:50,626 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-18'], host='arbutus.cloud-nowcast') 2024-04-18 03:47:50,631 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-17'], host='localhost') 2024-04-18 03:47:52,098 INFO [download_results] running in process 2023106 2024-04-18 03:47:52,099 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 03:47:52,100 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-18 03:47:52,102 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-18 03:47:52,102 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/17apr24 to /results/SalishSea/forecast2.202111 2024-04-18 03:49:47,495 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-04-18 04:05:46,002 INFO [archive_tarball] hindcast *apr19 results files archived to robot.graham 2024-04-18 04:05:46,005 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-04-18 04:12:34,355 INFO [download_results] forecast2 2024-04-17 results files from arbutus.cloud-nowcast downloaded 2024-04-18 04:12:34,360 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-18 04:12:34,582 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-04-17'], host='localhost') 2024-04-18 04:12:38,163 INFO [make_CHS_currents_file] running in process 2023254 2024-04-18 04:12:38,164 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 04:12:38,165 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-04-18 04:12:38,167 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-04-18 04:13:17,343 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-17 for forecast2 2024-04-18 04:13:17,369 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-04-18 04:13:17,547 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-04-17'], host='localhost') 2024-04-18 04:13:18,487 INFO [update_forecast_datasets] running in process 2023280 2024-04-18 04:13:18,488 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 04:13:18,489 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-04-18 04:13:18,490 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-18 04:13:18,492 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/17apr24 run 2024-04-18 04:17:47,907 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/17apr24 run: /results/SalishSea/rolling-forecasts/nemo 2024-04-18 04:17:47,908 INFO [update_forecast_datasets] nemo 2024-04-17 forecast2 rolling forecast datasets updated 2024-04-18 04:17:47,910 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-18 04:17:48,129 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-04-18 04:17:48,135 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-04-17'], host='localhost') 2024-04-18 04:17:48,140 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-04-17'], host='localhost') 2024-04-18 04:17:49,724 INFO [ping_erddap] running in process 2023355 2024-04-18 04:17:49,725 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 04:17:49,726 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-04-18 04:17:49,728 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-18 04:17:49,737 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-04-18 04:17:49,739 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-18 04:17:51,300 INFO [make_surface_current_tiles] running in process 2023358 2024-04-18 04:17:51,302 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 04:17:51,303 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2024-04-18 04:17:51,305 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-04-18 04:17:51,347 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-17 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/17apr24 2024-04-18 04:17:56,314 INFO [make_plots] running in process 2023357 2024-04-18 04:17:56,315 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 04:17:56,316 INFO [make_plots] publishing log messages to tcp://*:5564 2024-04-18 04:17:56,317 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-18 04:17:57,549 INFO [make_plots] preparing render list for 2024-04-17 NEMO forecast2 publish figures 2024-04-18 04:17:57,673 INFO [make_plots] added Website_thumbnail figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,674 INFO [make_plots] added Threshold_website figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,675 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,676 INFO [make_plots] added Boundary Bay figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,677 INFO [make_plots] added Campbell River figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,678 INFO [make_plots] added Cherry Point figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,680 INFO [make_plots] added Friday Harbor figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,681 INFO [make_plots] added Halfmoon Bay figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,681 INFO [make_plots] added Nanaimo figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,682 INFO [make_plots] added Neah Bay figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,683 INFO [make_plots] added New Westminster figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,684 INFO [make_plots] added Patricia Bay figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,685 INFO [make_plots] added Point Atkinson figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,686 INFO [make_plots] added Port Renfrew figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,688 INFO [make_plots] added Sand Heads figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,689 INFO [make_plots] added Sandy Cove figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,690 INFO [make_plots] added Squamish figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,692 INFO [make_plots] added Victoria figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,693 INFO [make_plots] added Woodwards Landing figure to 2024-04-17 NEMO forecast2 publish render list 2024-04-18 04:17:57,694 INFO [make_plots] starting to render nemo forecast2 publish 17apr24 figures 2024-04-18 04:32:48,868 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/17apr24 2024-04-18 04:36:37,377 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-04-18 04:36:37,378 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-04-18 04:36:38,526 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/17apr24 2024-04-18 04:36:38,534 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-17 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/17apr24 2024-04-18 04:36:38,535 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-17 forecast2 completed 2024-04-18 04:36:38,818 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-04-18 04:37:12,375 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:38:59,948 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:39:00,371 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:39:36,144 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:40:47,477 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:41:24,269 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:42:01,125 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:42:37,556 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:43:14,289 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:43:50,742 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:44:26,951 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:44:27,378 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:45:02,297 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:45:38,923 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-13 00:00:00Z to 2024-04-20 05:55:00Z 2024-04-18 04:46:15,093 INFO [make_plots] finished rendering nemo forecast2 publish 17apr24 figures 2024-04-18 04:46:15,096 INFO [make_plots] nemo publish plots for 2024-04-17 forecast2 completed 2024-04-18 04:46:15,102 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-18 04:46:15,449 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-04-17'], host='localhost') 2024-04-18 04:46:18,511 INFO [make_feeds] running in process 2025750 2024-04-18 04:46:18,512 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 04:46:18,513 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-04-18 04:46:18,515 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-04-18 04:46:46,314 INFO [make_feeds] ATOM feeds for 2024-04-17 forecast2 run completed 2024-04-18 04:46:46,318 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-04-18 04:46:46,674 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2024-04-18 04:46:47,438 INFO [clear_checklist] running in process 2025775 2024-04-18 04:46:47,440 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 04:46:47,441 INFO [clear_checklist] publishing log messages to tcp://*:5563 2024-04-18 04:46:47,442 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2024-04-18 04:46:47,443 INFO [clear_checklist] requesting that manager clear system state checklist 2024-04-18 04:46:47,446 INFO [manager] writing checklist to log file 2024-04-18 04:46:47,497 INFO [manager] checklist cleared 2024-04-18 04:46:47,500 INFO [clear_checklist] nowcast system checklist cleared 2024-04-18 04:46:47,502 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2024-04-18 04:46:48,401 INFO [rotate_logs] running in process 2025780 2024-04-18 04:46:48,403 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-18 04:46:48,404 INFO [rotate_logs] publishing log messages to tcp://*:5563 2024-04-18 04:46:48,405 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2024-04-18 04:46:48,406 INFO [rotate_logs] rotating log files