2024-04-15 04:51:07,785 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2024-04-15 04:51:07,792 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2024-04-15 04:51:07,797 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2024-04-15 04:51:07,799 INFO [rotate_logs] log files rotated 2024-04-15 04:51:07,804 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2024-04-15 06:15:56,675 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 06:15:56,905 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-08-26'], host='localhost') 2024-04-15 06:15:56,913 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 06:15:56,918 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 06:15:58,991 INFO [download_results] running in process 1971112 2024-04-15 06:15:58,994 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 06:15:58,995 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-15 06:15:58,996 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 06:15:58,997 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26aug18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 06:16:03,053 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 06:17:12,116 INFO [download_results] hindcast 2018-08-26 results files from optimum-hindcast downloaded 2024-04-15 06:17:12,132 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 06:17:12,199 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-08-26'], host='localhost') 2024-04-15 06:17:13,382 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 06:17:13,420 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-aug', 'robot.graham'], host='localhost') 2024-04-15 06:17:14,383 INFO [archive_tarball] running in process 1971165 2024-04-15 06:17:14,384 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 06:17:14,385 INFO [archive_tarball] publishing log messages to tcp://*:5564 2024-04-15 06:17:14,386 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-04-15 06:17:14,387 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-aug18.tar from /ocean/sallen/timbucktwo/oxygen/*aug18/ 2024-04-15 06:23:35,786 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-aug18.index from /ocean/dlatorne/oxygen-aug18.tar 2024-04-15 06:23:35,893 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-aug18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-04-15 06:32:54,639 INFO [archive_tarball] hindcast *aug18 results files archived to robot.graham 2024-04-15 06:32:54,641 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-04-15 07:46:07,589 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 07:46:07,810 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-01'], host='localhost') 2024-04-15 07:46:07,816 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 07:46:07,821 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 07:46:09,008 INFO [download_results] running in process 1971702 2024-04-15 07:46:09,009 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 07:46:09,011 INFO [download_results] publishing log messages to tcp://*:5566 2024-04-15 07:46:09,012 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 07:46:09,013 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01sep18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 07:46:13,070 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 07:47:05,872 INFO [download_results] hindcast 2018-09-01 results files from optimum-hindcast downloaded 2024-04-15 07:47:05,887 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 07:47:05,894 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-01'], host='localhost') 2024-04-15 07:47:07,121 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 09:02:55,359 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240415/12/ 2024-04-15 09:02:55,360 INFO [collect_weather] 2024-04-15 2.5km weather forecast 12 collection complete 2024-04-15 09:02:55,364 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-04-15 09:02:55,582 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-04-14'], host='localhost') 2024-04-15 09:02:55,588 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-04-14'], host='localhost') 2024-04-15 09:02:55,593 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-04-14'], host='localhost') 2024-04-15 09:02:55,598 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-04-14'], host='localhost') 2024-04-15 09:02:55,602 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2024-04-15 09:02:55,608 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2024-04-15 09:02:55,612 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2024-04-15 09:02:55,615 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2024-04-15 09:02:55,621 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2024-04-15 09:02:57,010 INFO [collect_NeahBay_ssh] running in process 1972880 2024-04-15 09:02:57,011 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:57,012 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562 2024-04-15 09:02:57,014 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:57,015 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-04-15 2024-04-15 09:02:57,016 INFO [collect_weather] running in process 1972882 2024-04-15 09:02:57,017 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:57,019 INFO [collect_weather] publishing log messages to tcp://*:5563 2024-04-15 09:02:57,020 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:57,028 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2024-04-15 09:02:57,297 INFO [download_live_ocean] running in process 1972881 2024-04-15 09:02:57,301 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:57,302 INFO [download_live_ocean] publishing log messages to tcp://*:5566 2024-04-15 09:02:57,303 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:57,304 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-04-15 2024-04-15 09:02:58,000 INFO [collect_river_data] running in process 1972876 2024-04-15 09:02:58,001 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:58,002 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-04-15 09:02:58,003 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:58,004 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-04-14 2024-04-15 09:02:58,011 INFO [collect_river_data] running in process 1972877 2024-04-15 09:02:58,012 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:58,013 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-04-15 09:02:58,014 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:58,015 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-04-14 2024-04-15 09:02:58,024 INFO [collect_river_data] running in process 1972874 2024-04-15 09:02:58,025 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:58,026 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-04-15 09:02:58,027 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:58,028 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-04-14 2024-04-15 09:02:58,031 INFO [collect_river_data] running in process 1972878 2024-04-15 09:02:58,033 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:58,034 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-04-15 09:02:58,035 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:58,036 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-04-14 2024-04-15 09:02:58,057 INFO [make_turbidity_file] running in process 1972879 2024-04-15 09:02:58,058 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:58,060 INFO [make_turbidity_file] publishing log messages to tcp://*:5571 2024-04-15 09:02:58,061 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:58,062 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-04-15 2024-04-15 09:02:58,236 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2024-04-14&endDT=2024-04-14 "HTTP/1.1 200 OK" 2024-04-15 09:02:58,238 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2024-04-14&endDT=2024-04-14 "HTTP/1.1 200 OK" 2024-04-15 09:02:58,239 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2024-04-14&endDT=2024-04-14 "HTTP/1.1 200 OK" 2024-04-15 09:02:58,241 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2024-04-14&endDT=2024-04-14 "HTTP/1.1 200 OK" 2024-04-15 09:02:58,346 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-04-14 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2024-04-15 09:02:58,347 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-04-14 completed 2024-04-15 09:02:58,351 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-04-14 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2024-04-15 09:02:58,352 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-04-14 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2024-04-15 09:02:58,353 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-04-14 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2024-04-15 09:02:58,354 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-15 09:02:58,355 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-04-14 completed 2024-04-15 09:02:58,356 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-04-14 completed 2024-04-15 09:02:58,357 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-04-14 completed 2024-04-15 09:02:58,364 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-15 09:02:58,378 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-15 09:02:58,393 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-15 09:02:58,484 INFO [collect_NeahBay_ssh] 2024-04-15T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2024-04-15 09:02:58,487 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-04-15 09:02:58,499 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 09:02:58,522 INFO [download_live_ocean] downloaded 10065755 bytes from https://liveocean.apl.uw.edu/output/f2024.04.15/ubc.nc to /results/forcing/LiveOcean/downloaded/20240415/low_passed_UBC.nc 2024-04-15 09:02:58,524 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2024-04-15 09:02:58,862 INFO [crop_gribs] running in process 1972883 2024-04-15 09:02:58,863 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:02:58,864 INFO [crop_gribs] publishing log messages to tcp://*:5562 2024-04-15 09:02:58,864 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:02:58,866 INFO [crop_gribs] cropping 2024-04-15 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2024-04-15 09:02:58,879 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240415/18/ 2024-04-15 09:02:59,568 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240415/low_passed_UBC.nc 2024-04-15 09:02:59,569 INFO [download_live_ocean] 2024-04-15 Live Ocean file for Salish Sea western boundary downloaded 2024-04-15 09:02:59,572 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2024-04-15 09:02:59,765 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-04-15'], host='localhost') 2024-04-15 09:03:01,223 INFO [make_turbidity_file] 2024-04-15 Fraser River turbidity file creation complete 2024-04-15 09:03:01,225 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2024-04-15 09:03:02,038 INFO [make_live_ocean_files] running in process 1973049 2024-04-15 09:03:02,039 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:03:02,040 INFO [make_live_ocean_files] publishing log messages to tcp://*:5566 2024-04-15 09:03:02,041 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:03:02,042 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-04-15 Live Ocean run 2024-04-15 09:03:02,067 INFO [make_ssh_files] running in process 1973030 2024-04-15 09:03:02,090 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:03:02,091 INFO [make_ssh_files] publishing log messages to tcp://*:5567 2024-04-15 09:03:02,092 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:03:02,093 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-04-15 NOAA Neah Bay 06Z observation and forecast values 2024-04-15 09:03:06,699 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d11.nc 2024-04-15 09:03:06,876 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d12.nc 2024-04-15 09:03:07,049 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d13.nc 2024-04-15 09:03:07,218 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d14.nc 2024-04-15 09:03:07,444 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d15.nc 2024-04-15 09:03:07,620 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d16.nc 2024-04-15 09:03:07,801 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc 2024-04-15 09:03:07,996 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d18.nc 2024-04-15 09:03:08,137 INFO [make_ssh_files] sea surface height boundary file for 2024-04-15 nowcast run created 2024-04-15 09:03:08,140 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-04-15 09:03:08,147 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-04-15 09:03:08,151 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-04-15 09:03:09,181 INFO [make_runoff_file] running in process 1973103 2024-04-15 09:03:09,182 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:03:09,183 INFO [make_runoff_file] publishing log messages to tcp://*:5567 2024-04-15 09:03:09,184 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:03:09,210 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-04-15 09:03:09,351 INFO [make_runoff_file] 2024-04-15 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-04-15 09:03:09,355 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-04-15 09:03:09,937 INFO [make_v202111_runoff_file] running in process 1973101 2024-04-15 09:03:09,938 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:03:09,939 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5567 2024-04-15 09:03:09,940 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:03:09,941 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-14 2024-04-15 09:03:12,859 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-14: /results/forcing/rivers/R202108Dailies_y2024m04d14.nc 2024-04-15 09:03:12,860 INFO [make_v202111_runoff_file] 2024-04-14 runoff file creation completed 2024-04-15 09:03:12,862 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-04-15 09:03:51,146 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: / 2024-04-15 09:03:51,147 INFO [make_live_ocean_files] 2024-04-15 Live Ocean western boundary conditions files created 2024-04-15 09:03:51,149 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2024-04-15 09:11:17,631 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 09:11:17,839 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-06'], host='localhost') 2024-04-15 09:11:17,845 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 09:11:17,852 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 09:11:19,027 INFO [download_results] running in process 1973167 2024-04-15 09:11:19,028 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:11:19,030 INFO [download_results] publishing log messages to tcp://*:5567 2024-04-15 09:11:19,031 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:11:19,032 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06sep18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 09:11:23,052 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 09:11:28,815 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240415/12/ 2024-04-15 09:11:28,817 INFO [crop_gribs] 2024-04-15 12 GRIBs cropping complete 2024-04-15 09:11:28,819 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-04-15 09:11:29,021 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2024-04-15 09:11:31,417 INFO [grib_to_netcdf] running in process 1973205 2024-04-15 09:11:31,419 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:11:31,420 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565 2024-04-15 09:11:31,421 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:11:31,423 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-15 nowcast and forecast runs 2024-04-15 09:11:45,834 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m04d15.nc 2024-04-15 09:11:57,557 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d16.nc 2024-04-15 09:12:04,154 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d17.nc 2024-04-15 09:12:04,176 INFO [grib_to_netcdf] 2024-04-15 NEMO-atmos forcing file for nowcast+ created 2024-04-15 09:12:04,179 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-04-15 09:12:04,195 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2024-04-15 09:12:04,199 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2024-04-15 09:12:04,202 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2024-04-15 09:12:04,205 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2024-04-15 09:12:04,211 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2024-04-15 09:12:05,221 INFO [ping_erddap] running in process 1973247 2024-04-15 09:12:05,223 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:12:05,224 INFO [ping_erddap] publishing log messages to tcp://*:5565 2024-04-15 09:12:05,225 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:12:05,226 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2024-04-15 09:12:05,229 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-15 09:12:05,377 INFO [upload_forcing] running in process 1973242 2024-04-15 09:12:05,378 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:12:05,380 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-04-15 09:12:05,381 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:12:05,386 INFO [upload_forcing] running in process 1973244 2024-04-15 09:12:05,387 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:12:05,389 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-04-15 09:12:05,390 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:12:05,432 INFO [upload_forcing] running in process 1973245 2024-04-15 09:12:05,434 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:12:05,435 INFO [upload_forcing] publishing log messages to tcp://*:5569 2024-04-15 09:12:05,437 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:12:05,438 INFO [upload_forcing] running in process 1973246 2024-04-15 09:12:05,439 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:12:05,441 INFO [upload_forcing] publishing log messages to tcp://*:5570 2024-04-15 09:12:05,442 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:12:08,204 INFO [upload_forcing] nowcast+ 2024-04-15 forcing files upload to optimum-hindcast completed 2024-04-15 09:12:08,209 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-15 09:12:08,358 INFO [upload_forcing] nowcast+ 2024-04-15 forcing files upload to orcinus-nowcast-agrif completed 2024-04-15 09:12:08,361 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-15 09:12:12,002 INFO [download_results] hindcast 2018-09-06 results files from optimum-hindcast downloaded 2024-04-15 09:12:12,019 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 09:12:12,037 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-06'], host='localhost') 2024-04-15 09:12:13,244 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 09:12:15,070 INFO [upload_forcing] nowcast+ 2024-04-15 forcing files upload to robot.graham completed 2024-04-15 09:12:15,075 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-15 09:12:19,526 INFO [upload_forcing] nowcast+ 2024-04-15 forcing files upload to arbutus.cloud-nowcast completed 2024-04-15 09:12:19,531 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-15 09:12:19,546 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 09:12:20,675 INFO [make_forcing_links] running in process 1973288 2024-04-15 09:12:20,677 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:12:20,678 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-04-15 09:12:20,680 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:12:21,872 INFO [make_forcing_links] nowcast+ 2024-04-15 forcing file links on arbutus.cloud-nowcast created 2024-04-15 09:12:21,877 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-15 09:12:21,890 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 09:12:24,026 INFO [run_NEMO] running in process 12797 2024-04-15 09:12:24,027 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:12:24,028 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-04-15 09:12:24,051 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:12:25,103 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/15apr24nowcast_2024-04-15T091224.083101-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-15 09:12:25,123 INFO [run_NEMO] nowcast NEMO run for 2024-04-15 on arbutus.cloud-nowcast started 2024-04-15 09:12:25,130 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-15 09:12:25,164 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-04-15 09:12:27,097 INFO [watch_NEMO] running in process 13059 2024-04-15 09:12:27,098 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:12:27,099 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-15 09:12:27,101 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:12:27,175 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-15 09:17:27,176 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17589336 = 2024-04-15 05:03:20 UTC, 21.1% complete 2024-04-15 09:22:27,177 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17589839 = 2024-04-15 10:38:40 UTC, 44.4% complete 2024-04-15 09:27:27,178 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17590310 = 2024-04-15 15:52:40 UTC, 66.2% complete 2024-04-15 09:32:27,180 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17590791 = 2024-04-15 21:13:20 UTC, 88.5% complete 2024-04-15 09:37:27,186 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2024-04-15 09:37:27,190 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-15 09:37:27,396 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 09:37:27,402 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 09:37:28,682 INFO [make_forcing_links] running in process 1973517 2024-04-15 09:37:28,683 INFO [download_results] running in process 1973518 2024-04-15 09:37:28,684 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:37:28,685 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-04-15 09:37:28,686 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:37:28,687 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:37:28,688 INFO [download_results] publishing log messages to tcp://*:5566 2024-04-15 09:37:28,689 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:37:28,690 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/15apr24 to /results/SalishSea/nowcast-blue.202111 2024-04-15 09:37:29,639 INFO [make_forcing_links] ssh 2024-04-15 forcing file links on arbutus.cloud-nowcast created 2024-04-15 09:37:29,644 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-15 09:37:29,679 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 09:37:31,765 INFO [run_NEMO] running in process 18325 2024-04-15 09:37:31,766 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:37:31,768 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-04-15 09:37:31,768 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:37:32,707 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/15apr24forecast_2024-04-15T093731.844911-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-15 09:37:32,728 INFO [run_NEMO] forecast NEMO run for 2024-04-15 on arbutus.cloud-nowcast started 2024-04-15 09:37:32,733 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-15 09:37:32,742 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-04-15 09:37:34,807 INFO [watch_NEMO] running in process 18582 2024-04-15 09:37:34,827 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:37:34,845 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-15 09:37:34,863 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:37:34,921 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-15 09:42:34,925 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17591470 = 2024-04-16 04:46:00 UTC, 13.2% complete 2024-04-15 09:47:34,926 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17591954 = 2024-04-16 10:08:40 UTC, 28.2% complete 2024-04-15 09:52:34,932 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17592430 = 2024-04-16 15:26:00 UTC, 42.9% complete 2024-04-15 09:57:34,933 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17592894 = 2024-04-16 20:35:20 UTC, 57.2% complete 2024-04-15 09:57:42,057 INFO [download_results] nowcast 2024-04-15 results files from arbutus.cloud-nowcast downloaded 2024-04-15 09:57:42,062 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 09:57:42,077 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 09:57:42,082 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-04-14'], host='localhost') 2024-04-15 09:57:42,087 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 09:57:44,510 INFO [make_CHS_currents_file] running in process 1973601 2024-04-15 09:57:44,511 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:57:44,512 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565 2024-04-15 09:57:44,513 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:57:48,248 INFO [make_plots] running in process 1973600 2024-04-15 09:57:48,249 INFO [make_plots] running in process 1973598 2024-04-15 09:57:48,250 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:57:48,252 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 09:57:48,252 INFO [make_plots] publishing log messages to tcp://*:5566 2024-04-15 09:57:48,253 INFO [make_plots] publishing log messages to tcp://*:5567 2024-04-15 09:57:48,254 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:57:48,255 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 09:57:49,078 INFO [make_plots] preparing render list for 2024-04-15 NEMO nowcast-blue research figures 2024-04-15 09:57:49,080 INFO [make_plots] preparing render list for 2024-04-14 NEMO nowcast-blue comparison figures 2024-04-15 09:57:50,566 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-blue research render list 2024-04-15 09:57:51,666 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-blue research render list 2024-04-15 09:57:51,666 INFO [make_plots] added Currents_sections_and_surface figure to 2024-04-15 NEMO nowcast-blue research render list 2024-04-15 09:57:51,667 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-04-15 NEMO nowcast-blue research render list 2024-04-15 09:57:51,668 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-04-15 NEMO nowcast-blue research render list 2024-04-15 09:57:51,669 INFO [make_plots] starting to render nemo nowcast research 15apr24 figures 2024-04-15 09:58:11,131 INFO [make_plots] added SH_wind figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,132 INFO [make_plots] added Compare_VENUS_East figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,133 INFO [make_plots] added Compare_VENUS_Central figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,133 INFO [make_plots] added Central_ADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,134 INFO [make_plots] added Central_depavADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,135 INFO [make_plots] added Central_timeavADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,136 INFO [make_plots] added East_ADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,137 INFO [make_plots] added East_depavADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,138 INFO [make_plots] added East_timeavADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,138 INFO [make_plots] added ddl_ADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,139 INFO [make_plots] added ddl_depavADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,139 INFO [make_plots] added ddl_timeavADCP figure to 2024-04-14 NEMO nowcast-blue comparison render list 2024-04-15 09:58:11,140 INFO [make_plots] starting to render nemo nowcast comparison 14apr24 figures 2024-04-15 09:58:21,478 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-15 for nowcast 2024-04-15 09:58:21,481 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-04-15 09:59:12,674 INFO [make_plots] finished rendering nemo nowcast research 15apr24 figures 2024-04-15 09:59:12,677 INFO [make_plots] nemo research plots for 2024-04-15 nowcast completed 2024-04-15 09:59:12,698 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 10:00:19,321 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-04-15 10:00:19,864 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-04-15 10:00:20,403 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-04-15 10:00:20,937 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-04-15 10:00:21,473 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-04-15 10:00:22,007 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-04-15 10:00:22,542 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-04-15 10:00:23,069 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-04-15 10:00:23,597 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-04-15 10:00:23,600 INFO [make_plots] finished rendering nemo nowcast comparison 14apr24 figures 2024-04-15 10:00:23,717 INFO [make_plots] nemo comparison plots for 2024-04-14 nowcast completed 2024-04-15 10:00:23,720 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 10:02:34,922 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17593353 = 2024-04-17 01:41:20 UTC, 71.4% complete 2024-04-15 10:07:34,922 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17593837 = 2024-04-17 07:04:00 UTC, 86.3% complete 2024-04-15 10:12:34,923 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17594280 = 2024-04-17 11:59:20 UTC, 100.0% complete 2024-04-15 10:17:34,927 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2024-04-15 10:17:34,933 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-15 10:17:35,140 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 10:17:35,147 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 10:17:35,150 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:17:35,157 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:17:35,162 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:17:35,166 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:17:35,170 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:17:36,403 INFO [upload_forcing] running in process 1974124 2024-04-15 10:17:36,405 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:17:36,406 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-04-15 10:17:36,407 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:17:36,408 INFO [upload_forcing] running in process 1974121 2024-04-15 10:17:36,409 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:17:36,410 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-04-15 10:17:36,411 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:17:36,416 INFO [upload_forcing] running in process 1974123 2024-04-15 10:17:36,417 INFO [upload_forcing] running in process 1974122 2024-04-15 10:17:36,418 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:17:36,419 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:17:36,421 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-04-15 10:17:36,421 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-04-15 10:17:36,422 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:17:36,423 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:17:36,430 INFO [download_results] running in process 1974125 2024-04-15 10:17:36,431 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:17:36,432 INFO [download_results] publishing log messages to tcp://*:5569 2024-04-15 10:17:36,433 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:17:36,435 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/15apr24 to /results/SalishSea/forecast.202111 2024-04-15 10:17:36,549 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-04-15 10:17:36,850 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:17:36,852 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner 2024-04-15 10:17:36,856 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-04-15 10:17:36,858 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:17:36,861 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline 2024-04-15 10:17:36,863 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-04-15 10:17:36,866 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:17:36,869 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout 2024-04-15 10:17:36,871 ERROR [upload_forcing] raise EOFError() 2024-04-15 10:17:36,872 ERROR [upload_forcing] EOFError 2024-04-15 10:17:36,873 ERROR [upload_forcing] 2024-04-15 10:17:36,875 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-04-15 10:17:36,876 ERROR [upload_forcing] 2024-04-15 10:17:36,878 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:17:36,888 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run 2024-04-15 10:17:36,889 ERROR [upload_forcing] self._check_banner() 2024-04-15 10:17:36,899 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner 2024-04-15 10:17:36,916 ERROR [upload_forcing] raise SSHException( 2024-04-15 10:17:36,934 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:17:36,952 ERROR [upload_forcing] 2024-04-15 10:17:36,957 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-04-15 10:17:36,958 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:17:36,960 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner 2024-04-15 10:17:36,961 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-04-15 10:17:36,961 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:17:36,962 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline 2024-04-15 10:17:36,963 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-04-15 10:17:36,964 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:17:36,966 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout 2024-04-15 10:17:36,967 ERROR [upload_forcing] raise EOFError() 2024-04-15 10:17:36,968 ERROR [upload_forcing] EOFError 2024-04-15 10:17:36,968 ERROR [upload_forcing] 2024-04-15 10:17:36,969 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-04-15 10:17:36,970 ERROR [upload_forcing] 2024-04-15 10:17:36,971 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:17:36,972 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run 2024-04-15 10:17:36,973 ERROR [upload_forcing] self._check_banner() 2024-04-15 10:17:36,974 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner 2024-04-15 10:17:36,975 ERROR [upload_forcing] raise SSHException( 2024-04-15 10:17:36,976 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:17:36,977 ERROR [upload_forcing] 2024-04-15 10:17:36,978 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner buf = self.packetizer.readline(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline buf += self._read_timeout(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout raise EOFError() EOFError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/client.py", line 430, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 704, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 96, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 85, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/client.py", line 430, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 704, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:17:37,261 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-15 10:17:37,264 INFO [upload_forcing] turbidity 2024-04-15 forcing files upload to optimum-hindcast completed 2024-04-15 10:17:38,055 INFO [upload_forcing] turbidity 2024-04-15 forcing files upload to arbutus.cloud-nowcast completed 2024-04-15 10:17:38,059 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-15 10:17:38,081 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:17:39,396 INFO [make_forcing_links] running in process 1974162 2024-04-15 10:17:39,409 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:17:39,410 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-04-15 10:17:39,411 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:17:40,441 INFO [upload_forcing] turbidity 2024-04-15 forcing files upload to robot.graham completed 2024-04-15 10:17:40,445 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-15 10:17:40,885 INFO [make_forcing_links] nowcast-green 2024-04-15 forcing file links on arbutus.cloud-nowcast created 2024-04-15 10:17:40,887 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-15 10:17:40,907 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 10:17:42,237 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-04-15 10:17:44,210 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/15apr24nowcast-green_2024-04-15T101743.266621-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-15 10:17:44,220 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-15 10:17:44,237 INFO [run_NEMO] nowcast-green NEMO run for 2024-04-15 on arbutus.cloud-nowcast started 2024-04-15 10:17:44,239 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2024-04-15 10:17:46,448 INFO [watch_NEMO] running in process 25356 2024-04-15 10:17:46,449 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:17:46,450 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-15 10:17:46,451 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:17:46,575 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-15 10:18:21,662 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-04-15 10:18:21,705 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 10:18:23,912 INFO [run_ww3] running in process 25480 2024-04-15 10:18:23,913 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:18:23,914 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-04-15 10:18:23,915 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:18:23,920 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-15T101823.897988-0700 2024-04-15 10:18:23,930 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-15T101823.897988-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-04-15 10:18:23,937 INFO [run_ww3] nowcast WaveWatch3 run for 2024-04-15 on arbutus.cloud-nowcast started 2024-04-15 10:18:23,963 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-15 10:18:23,986 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-04-15 10:18:26,778 INFO [watch_ww3] running in process 25570 2024-04-15 10:18:26,782 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:18:26,783 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-04-15 10:18:26,784 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:18:26,863 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-15 10:22:46,574 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17589022 = 2024-04-15 01:34:00 UTC, 6.5% complete 2024-04-15 10:23:26,866 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete 2024-04-15 10:27:31,419 INFO [upload_forcing] running in process 1974507 2024-04-15 10:27:31,421 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:27:31,422 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-04-15 10:27:31,423 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:27:31,491 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-04-15 10:27:31,947 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:27:31,949 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner 2024-04-15 10:27:31,950 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-04-15 10:27:31,952 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:27:31,954 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline 2024-04-15 10:27:31,956 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-04-15 10:27:31,957 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:27:31,959 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout 2024-04-15 10:27:31,961 ERROR [upload_forcing] raise EOFError() 2024-04-15 10:27:31,962 ERROR [upload_forcing] EOFError 2024-04-15 10:27:31,964 ERROR [upload_forcing] 2024-04-15 10:27:31,966 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-04-15 10:27:31,967 ERROR [upload_forcing] 2024-04-15 10:27:31,969 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:27:31,971 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run 2024-04-15 10:27:31,972 ERROR [upload_forcing] self._check_banner() 2024-04-15 10:27:31,974 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner 2024-04-15 10:27:31,976 ERROR [upload_forcing] raise SSHException( 2024-04-15 10:27:31,978 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:27:31,979 ERROR [upload_forcing] 2024-04-15 10:27:31,989 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-04-15 10:27:31,991 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:27:31,993 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner 2024-04-15 10:27:31,994 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-04-15 10:27:31,996 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:27:31,998 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline 2024-04-15 10:27:32,000 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-04-15 10:27:32,002 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:27:32,003 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout 2024-04-15 10:27:32,005 ERROR [upload_forcing] raise EOFError() 2024-04-15 10:27:32,006 ERROR [upload_forcing] EOFError 2024-04-15 10:27:32,008 ERROR [upload_forcing] 2024-04-15 10:27:32,010 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-04-15 10:27:32,012 ERROR [upload_forcing] 2024-04-15 10:27:32,013 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:27:32,015 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run 2024-04-15 10:27:32,017 ERROR [upload_forcing] self._check_banner() 2024-04-15 10:27:32,019 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner 2024-04-15 10:27:32,021 ERROR [upload_forcing] raise SSHException( 2024-04-15 10:27:32,023 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:27:32,024 ERROR [upload_forcing] 2024-04-15 10:27:32,031 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner buf = self.packetizer.readline(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline buf += self._read_timeout(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout raise EOFError() EOFError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/client.py", line 430, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 704, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 96, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 85, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/client.py", line 430, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 704, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:27:46,578 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17589197 = 2024-04-15 03:30:40 UTC, 14.6% complete 2024-04-15 10:27:51,797 INFO [upload_forcing] running in process 1974519 2024-04-15 10:27:51,798 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:27:51,799 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-04-15 10:27:51,801 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:27:51,857 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-04-15 10:27:52,068 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:27:52,070 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner 2024-04-15 10:27:52,071 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-04-15 10:27:52,073 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:27:52,076 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline 2024-04-15 10:27:52,077 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-04-15 10:27:52,079 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:27:52,097 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout 2024-04-15 10:27:52,115 ERROR [upload_forcing] raise EOFError() 2024-04-15 10:27:52,134 ERROR [upload_forcing] EOFError 2024-04-15 10:27:52,143 ERROR [upload_forcing] 2024-04-15 10:27:52,144 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-04-15 10:27:52,145 ERROR [upload_forcing] 2024-04-15 10:27:52,146 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:27:52,148 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run 2024-04-15 10:27:52,149 ERROR [upload_forcing] self._check_banner() 2024-04-15 10:27:52,150 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner 2024-04-15 10:27:52,151 ERROR [upload_forcing] raise SSHException( 2024-04-15 10:27:52,152 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:27:52,153 ERROR [upload_forcing] 2024-04-15 10:27:52,154 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-04-15 10:27:52,154 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:27:52,155 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner 2024-04-15 10:27:52,156 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-04-15 10:27:52,157 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:27:52,157 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline 2024-04-15 10:27:52,158 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-04-15 10:27:52,159 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:27:52,160 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout 2024-04-15 10:27:52,160 ERROR [upload_forcing] raise EOFError() 2024-04-15 10:27:52,161 ERROR [upload_forcing] EOFError 2024-04-15 10:27:52,162 ERROR [upload_forcing] 2024-04-15 10:27:52,163 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-04-15 10:27:52,163 ERROR [upload_forcing] 2024-04-15 10:27:52,164 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:27:52,165 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run 2024-04-15 10:27:52,165 ERROR [upload_forcing] self._check_banner() 2024-04-15 10:27:52,166 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner 2024-04-15 10:27:52,167 ERROR [upload_forcing] raise SSHException( 2024-04-15 10:27:52,167 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:27:52,168 ERROR [upload_forcing] 2024-04-15 10:27:52,169 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner buf = self.packetizer.readline(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline buf += self._read_timeout(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout raise EOFError() EOFError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/client.py", line 430, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 704, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 96, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 85, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/client.py", line 430, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 704, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:28:26,875 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete 2024-04-15 10:32:46,574 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17589372 = 2024-04-15 05:27:20 UTC, 22.7% complete 2024-04-15 10:33:26,870 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 291, 67.4% complete 2024-04-15 10:36:27,641 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 10:36:27,854 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-11'], host='localhost') 2024-04-15 10:36:27,922 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 10:36:27,928 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 10:36:29,179 INFO [download_results] running in process 1974608 2024-04-15 10:36:29,180 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:36:29,182 INFO [download_results] publishing log messages to tcp://*:5565 2024-04-15 10:36:29,183 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:36:29,184 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11sep18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 10:36:33,237 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 10:37:30,879 INFO [download_results] hindcast 2018-09-11 results files from optimum-hindcast downloaded 2024-04-15 10:37:30,889 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 10:37:30,908 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-11'], host='localhost') 2024-04-15 10:37:32,267 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 10:37:46,575 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17589547 = 2024-04-15 07:24:00 UTC, 30.8% complete 2024-04-15 10:38:26,875 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 396, 91.7% complete 2024-04-15 10:42:46,573 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17589722 = 2024-04-15 09:20:40 UTC, 39.0% complete 2024-04-15 10:43:26,903 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-15 10:43:26,905 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-15 10:43:27,128 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:43:27,134 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 10:43:28,167 INFO [download_wwatch3_results] running in process 1974699 2024-04-15 10:43:28,169 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:43:28,170 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565 2024-04-15 10:43:28,171 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:43:29,571 INFO [run_ww3] forecast WaveWatch3 run for 2024-04-15 on arbutus.cloud-nowcast started 2024-04-15 10:43:29,674 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-15 10:43:29,699 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-04-15 10:43:32,185 INFO [watch_ww3] running in process 28699 2024-04-15 10:43:32,187 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:43:32,188 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-04-15 10:43:32,189 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:43:32,479 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-15 10:44:47,542 INFO [upload_forcing] running in process 1974711 2024-04-15 10:44:47,567 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:44:47,586 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-04-15 10:44:47,600 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:44:47,653 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-04-15 10:44:47,951 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:44:47,953 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner 2024-04-15 10:44:47,956 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-04-15 10:44:47,961 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:44:47,962 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline 2024-04-15 10:44:47,964 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-04-15 10:44:47,966 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:44:47,967 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout 2024-04-15 10:44:47,969 ERROR [upload_forcing] raise EOFError() 2024-04-15 10:44:47,970 ERROR [upload_forcing] EOFError 2024-04-15 10:44:47,972 ERROR [upload_forcing] 2024-04-15 10:44:47,973 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-04-15 10:44:47,975 ERROR [upload_forcing] 2024-04-15 10:44:47,976 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:44:47,978 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run 2024-04-15 10:44:47,980 ERROR [upload_forcing] self._check_banner() 2024-04-15 10:44:47,982 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner 2024-04-15 10:44:47,983 ERROR [upload_forcing] raise SSHException( 2024-04-15 10:44:47,985 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:44:47,987 ERROR [upload_forcing] 2024-04-15 10:44:47,997 ERROR [upload_forcing] Exception (client): Error reading SSH protocol banner 2024-04-15 10:44:47,999 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:44:48,000 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner 2024-04-15 10:44:48,002 ERROR [upload_forcing] buf = self.packetizer.readline(timeout) 2024-04-15 10:44:48,004 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:44:48,006 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline 2024-04-15 10:44:48,007 ERROR [upload_forcing] buf += self._read_timeout(timeout) 2024-04-15 10:44:48,008 ERROR [upload_forcing] ^^^^^^^^^^^^^^^^^^^^^^^^^^^ 2024-04-15 10:44:48,010 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout 2024-04-15 10:44:48,012 ERROR [upload_forcing] raise EOFError() 2024-04-15 10:44:48,013 ERROR [upload_forcing] EOFError 2024-04-15 10:44:48,015 ERROR [upload_forcing] 2024-04-15 10:44:48,016 ERROR [upload_forcing] During handling of the above exception, another exception occurred: 2024-04-15 10:44:48,018 ERROR [upload_forcing] 2024-04-15 10:44:48,020 ERROR [upload_forcing] Traceback (most recent call last): 2024-04-15 10:44:48,021 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run 2024-04-15 10:44:48,023 ERROR [upload_forcing] self._check_banner() 2024-04-15 10:44:48,025 ERROR [upload_forcing] File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner 2024-04-15 10:44:48,027 ERROR [upload_forcing] raise SSHException( 2024-04-15 10:44:48,029 ERROR [upload_forcing] paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:44:48,031 ERROR [upload_forcing] 2024-04-15 10:44:48,056 CRITICAL [upload_forcing] unhandled exception: Traceback (most recent call last): File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2270, in _check_banner buf = self.packetizer.readline(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 374, in readline buf += self._read_timeout(timeout) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/packet.py", line 603, in _read_timeout raise EOFError() EOFError During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 76, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/client.py", line 430, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 704, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work checklist = self.worker_func( ^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/upload_forcing.py", line 96, in upload_forcing ssh_client, sftp_client = ssh_sftp.sftp(host_name, ssh_key) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 145, in sftp ssh_client = ssh(host, key_filename, ssh_config_file) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/SalishSeaCast/SalishSeaNowcast/nowcast/ssh_sftp.py", line 85, in ssh ssh_client.connect( File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/client.py", line 430, in connect t.start_client(timeout=timeout) File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 704, in start_client raise e File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2093, in run self._check_banner() File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/paramiko/transport.py", line 2274, in _check_banner raise SSHException( paramiko.ssh_exception.SSHException: Error reading SSH protocol banner 2024-04-15 10:47:46,575 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17589898 = 2024-04-15 11:18:00 UTC, 47.1% complete 2024-04-15 10:48:32,429 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete 2024-04-15 10:48:43,589 INFO [download_results] forecast 2024-04-15 results files from arbutus.cloud-nowcast downloaded 2024-04-15 10:48:43,594 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 10:48:43,614 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:48:47,446 INFO [make_CHS_currents_file] running in process 1974731 2024-04-15 10:48:47,447 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:48:47,449 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5566 2024-04-15 10:48:47,450 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:49:20,277 INFO [download_wwatch3_results] nowcast 2024-04-15 results files from arbutus.cloud-nowcast downloaded 2024-04-15 10:49:20,280 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-15 10:49:34,737 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-15 for forecast 2024-04-15 10:49:34,740 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-04-15 10:49:34,755 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:49:35,781 INFO [update_forecast_datasets] running in process 1974758 2024-04-15 10:49:35,784 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:49:35,785 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565 2024-04-15 10:49:35,787 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:49:35,788 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/15apr24 run 2024-04-15 10:49:36,050 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/15apr24 run: /results/SalishSea/rolling-forecasts/nemo 2024-04-15 10:49:36,051 INFO [update_forecast_datasets] nemo 2024-04-15 forecast rolling forecast datasets updated 2024-04-15 10:49:36,053 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-15 10:49:36,071 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-04-15 10:49:36,075 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:49:36,080 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 10:49:37,232 INFO [ping_erddap] running in process 1974763 2024-04-15 10:49:37,233 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:49:37,234 INFO [ping_erddap] publishing log messages to tcp://*:5565 2024-04-15 10:49:37,235 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:49:37,246 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-04-15 10:49:37,247 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-15 10:49:38,956 INFO [make_surface_current_tiles] running in process 1974766 2024-04-15 10:49:38,957 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:49:38,958 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565 2024-04-15 10:49:38,959 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:49:38,991 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-15 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/15apr24 2024-04-15 10:49:42,258 INFO [make_plots] running in process 1974765 2024-04-15 10:49:42,281 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 10:49:42,290 INFO [make_plots] publishing log messages to tcp://*:5566 2024-04-15 10:49:42,290 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 10:49:43,478 INFO [make_plots] preparing render list for 2024-04-15 NEMO forecast publish figures 2024-04-15 10:49:43,729 INFO [make_plots] added Website_thumbnail figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,731 INFO [make_plots] added Threshold_website figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,732 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,733 INFO [make_plots] added Boundary Bay figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,734 INFO [make_plots] added Campbell River figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,735 INFO [make_plots] added Cherry Point figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,735 INFO [make_plots] added Friday Harbor figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,737 INFO [make_plots] added Halfmoon Bay figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,737 INFO [make_plots] added Nanaimo figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,739 INFO [make_plots] added Neah Bay figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,740 INFO [make_plots] added New Westminster figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,741 INFO [make_plots] added Patricia Bay figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,742 INFO [make_plots] added Point Atkinson figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,742 INFO [make_plots] added Port Renfrew figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,743 INFO [make_plots] added Sand Heads figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,744 INFO [make_plots] added Sandy Cove figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,745 INFO [make_plots] added Squamish figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,746 INFO [make_plots] added Victoria figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,747 INFO [make_plots] added Woodwards Landing figure to 2024-04-15 NEMO forecast publish render list 2024-04-15 10:49:43,748 INFO [make_plots] starting to render nemo forecast publish 15apr24 figures 2024-04-15 10:52:46,560 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17590073 = 2024-04-15 13:14:40 UTC, 55.2% complete 2024-04-15 10:53:32,410 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete 2024-04-15 10:57:46,560 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17590248 = 2024-04-15 15:11:20 UTC, 63.3% complete 2024-04-15 10:58:32,410 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 300, 46.3% complete 2024-04-15 11:02:46,560 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17590423 = 2024-04-15 17:08:00 UTC, 71.4% complete 2024-04-15 11:03:32,410 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 408, 63.0% complete 2024-04-15 11:04:09,263 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/15apr24 2024-04-15 11:07:46,560 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17590598 = 2024-04-15 19:04:40 UTC, 79.5% complete 2024-04-15 11:08:19,353 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/15apr24 2024-04-15 11:08:19,362 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-15 forecast into /results/nowcast-sys/figures/surface_currents/forecast/15apr24 2024-04-15 11:08:19,363 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-15 forecast completed 2024-04-15 11:08:19,690 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-04-15 11:08:28,953 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-04-15 11:08:28,954 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-04-15 11:08:32,410 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 513, 79.2% complete 2024-04-15 11:09:06,343 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:11:00,193 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:11:00,619 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:11:38,603 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:12:46,561 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17590774 = 2024-04-15 21:02:00 UTC, 87.7% complete 2024-04-15 11:12:54,048 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:13:32,410 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 615, 94.9% complete 2024-04-15 11:13:32,757 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:14:10,616 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:14:48,270 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:15:25,140 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:16:04,094 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:16:43,028 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:16:43,437 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:17:21,496 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:17:46,561 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17590949 = 2024-04-15 22:58:40 UTC, 95.8% complete 2024-04-15 11:17:59,964 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-10 00:00:00Z to 2024-04-17 11:55:00Z 2024-04-15 11:18:32,411 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-15 11:18:32,415 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-15 11:18:32,820 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 11:18:34,118 INFO [download_wwatch3_results] running in process 1976895 2024-04-15 11:18:34,119 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:18:34,120 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565 2024-04-15 11:18:34,121 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:18:37,622 INFO [make_plots] finished rendering nemo forecast publish 15apr24 figures 2024-04-15 11:18:37,625 INFO [make_plots] nemo publish plots for 2024-04-15 forecast completed 2024-04-15 11:18:37,634 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 11:18:37,812 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 11:18:42,145 INFO [make_feeds] running in process 1976903 2024-04-15 11:18:42,147 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:18:42,148 INFO [make_feeds] publishing log messages to tcp://*:5566 2024-04-15 11:18:42,149 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:19:12,575 INFO [make_feeds] ATOM feeds for 2024-04-15 forecast run completed 2024-04-15 11:19:12,578 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-04-15 11:21:13,035 INFO [download_wwatch3_results] forecast 2024-04-15 results files from arbutus.cloud-nowcast downloaded 2024-04-15 11:21:13,039 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-15 11:21:13,224 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 11:21:14,284 INFO [update_forecast_datasets] running in process 1976932 2024-04-15 11:21:14,285 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:21:14,286 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565 2024-04-15 11:21:14,287 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:21:14,289 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/15apr24 run 2024-04-15 11:21:14,291 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/15apr24 run 2024-04-15 11:21:14,355 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/15apr24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-04-15 11:21:14,357 INFO [update_forecast_datasets] wwatch3 2024-04-15 forecast rolling forecast datasets updated 2024-04-15 11:21:14,360 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-15 11:21:14,543 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-04-15 11:21:15,455 INFO [ping_erddap] running in process 1976937 2024-04-15 11:21:15,456 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:21:15,458 INFO [ping_erddap] publishing log messages to tcp://*:5565 2024-04-15 11:21:15,459 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:21:15,461 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-04-15 11:21:15,463 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-15 11:21:15,643 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 11:21:19,637 INFO [make_plots] running in process 1976942 2024-04-15 11:21:19,638 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:21:19,639 INFO [make_plots] publishing log messages to tcp://*:5565 2024-04-15 11:21:19,641 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:21:19,642 INFO [make_plots] preparing render list for 2024-04-15 WaveWatch3 forecast publish figures 2024-04-15 11:21:19,643 INFO [make_plots] added Halibut Bank figure to 2024-04-15 WaveWatch3 forecast publish render list 2024-04-15 11:21:19,644 INFO [make_plots] added Sentry Shoal figure to 2024-04-15 WaveWatch3 forecast publish render list 2024-04-15 11:21:19,644 INFO [make_plots] starting to render wwatch3 forecast publish 15apr24 figures 2024-04-15 11:21:19,869 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-04-15 11:21:27,985 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-04-15 11:21:33,360 INFO [make_plots] finished rendering wwatch3 forecast publish 15apr24 figures 2024-04-15 11:21:33,361 INFO [make_plots] wwatch3 publish plots for 2024-04-15 forecast completed 2024-04-15 11:21:33,366 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 11:22:46,562 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17591040 = 2024-04-15 23:59:20 UTC, 100.0% complete 2024-04-15 11:24:50,205 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-04-15 11:25:43,272 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-04-15 11:25:43,461 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-13'], host='arbutus.cloud-nowcast') 2024-04-15 11:25:45,517 INFO [run_ww3] running in process 3203 2024-04-15 11:25:45,540 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:25:45,541 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-04-15 11:25:45,542 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:25:45,547 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-15T112545.515795-0700 2024-04-15 11:25:45,550 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-15T112545.515795-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-04-15 11:25:45,553 INFO [run_ww3] nowcast WaveWatch3 run for 2024-04-13 on arbutus.cloud-nowcast started 2024-04-15 11:25:45,556 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-15 11:25:45,752 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-04-15 11:25:47,776 INFO [watch_ww3] running in process 3297 2024-04-15 11:25:47,793 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:25:47,794 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-04-15 11:25:47,795 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:25:47,798 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-15 11:27:11,207 INFO [upload_forcing] running in process 1976989 2024-04-15 11:27:11,209 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:27:11,210 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-04-15 11:27:11,211 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:27:12,206 INFO [upload_forcing] turbidity 2024-04-15 forcing files upload to orcinus-nowcast-agrif completed 2024-04-15 11:27:12,210 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-15 11:27:12,394 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 11:27:13,591 INFO [make_forcing_links] running in process 1976997 2024-04-15 11:27:13,592 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:27:13,593 INFO [make_forcing_links] publishing log messages to tcp://*:5565 2024-04-15 11:27:13,594 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:27:14,536 INFO [make_forcing_links] nowcast-agrif 2024-04-15 forcing file links on orcinus-nowcast-agrif created 2024-04-15 11:27:14,541 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-15 11:27:14,736 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 11:27:16,023 INFO [run_NEMO_agrif] running in process 1977006 2024-04-15 11:27:16,027 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:27:16,029 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5565 2024-04-15 11:27:16,030 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:27:42,572 INFO [run_NEMO_agrif] job id for 15apr24nowcast-agrif: 11091780.orca2.ibb 2024-04-15 11:27:42,573 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-04-15 queued on orcinus-nowcast-agrif 2024-04-15 11:27:42,578 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker 2024-04-15 11:27:42,766 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11091780.orca2.ibb'], host='localhost') 2024-04-15 11:27:43,959 INFO [watch_NEMO_agrif] running in process 1977021 2024-04-15 11:27:43,979 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:27:43,994 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5565 2024-04-15 11:27:43,995 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:27:44,707 INFO [watch_NEMO_agrif] watching 15apr24nowcast-agrif job 11091780 on orcinus-nowcast-agrif 2024-04-15 11:27:44,893 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif job 11091780 is queued on orcinus-nowcast-agrif 2024-04-15 11:27:46,569 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2024-04-15 11:27:46,608 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-15 11:27:46,942 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 11:27:48,313 INFO [download_results] running in process 1977030 2024-04-15 11:27:48,314 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:27:48,315 INFO [download_results] publishing log messages to tcp://*:5566 2024-04-15 11:27:48,325 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:27:48,343 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/15apr24 to /results2/SalishSea/nowcast-green.202111 2024-04-15 11:28:45,962 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2024-04-15 11:30:47,863 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete 2024-04-15 11:33:46,411 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7609872 = 2024-04-15 02:07:20 UTC, 8.8% complete 2024-04-15 11:35:47,911 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete 2024-04-15 11:38:46,792 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7610082 = 2024-04-15 04:27:20 UTC, 18.6% complete 2024-04-15 11:40:47,814 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 300, 69.4% complete 2024-04-15 11:43:47,270 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7610293 = 2024-04-15 06:48:00 UTC, 28.3% complete 2024-04-15 11:45:47,809 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 399, 92.4% complete 2024-04-15 11:48:47,709 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7610503 = 2024-04-15 09:08:00 UTC, 38.1% complete 2024-04-15 11:50:47,794 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-15 11:50:47,809 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-15 11:50:48,274 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-13'], host='localhost') 2024-04-15 11:50:48,280 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-13'], host='arbutus.cloud-nowcast') 2024-04-15 11:50:49,440 INFO [download_wwatch3_results] running in process 1977113 2024-04-15 11:50:49,441 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:50:49,442 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567 2024-04-15 11:50:49,443 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:50:50,705 INFO [run_ww3] running in process 7311 2024-04-15 11:50:50,706 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 11:50:50,707 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-04-15 11:50:50,708 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 11:50:50,711 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-15T115050.704927-0700 2024-04-15 11:50:50,724 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-15T115050.704927-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-04-15 11:50:50,733 INFO [run_ww3] forecast WaveWatch3 run for 2024-04-13 on arbutus.cloud-nowcast started 2024-04-15 11:50:50,760 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-15 11:50:50,947 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-04-15 11:50:54,071 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-15 11:53:48,075 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7610714 = 2024-04-15 11:28:40 UTC, 47.8% complete 2024-04-15 11:55:54,969 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete 2024-04-15 11:57:09,614 INFO [download_wwatch3_results] nowcast 2024-04-13 results files from arbutus.cloud-nowcast downloaded 2024-04-15 11:57:09,617 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-15 11:58:48,621 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7610925 = 2024-04-15 13:49:20 UTC, 57.6% complete 2024-04-15 12:00:54,130 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete 2024-04-15 12:01:38,163 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 12:01:38,559 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-16'], host='localhost') 2024-04-15 12:01:38,564 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 12:01:38,569 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 12:01:39,904 INFO [download_results] running in process 1977196 2024-04-15 12:01:39,927 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:01:39,945 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-15 12:01:39,963 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:01:39,968 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16sep18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 12:01:44,004 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 12:02:55,206 INFO [download_results] hindcast 2018-09-16 results files from optimum-hindcast downloaded 2024-04-15 12:02:55,221 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 12:02:55,404 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-16'], host='localhost') 2024-04-15 12:02:56,446 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 12:03:48,972 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7611134 = 2024-04-15 16:08:40 UTC, 67.3% complete 2024-04-15 12:03:58,930 INFO [download_results] nowcast-green 2024-04-15 results files from arbutus.cloud-nowcast downloaded 2024-04-15 12:03:58,935 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 12:03:59,135 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 12:03:59,142 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2024-04-15 12:03:59,147 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 12:03:59,152 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 12:03:59,156 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 12:04:00,431 INFO [ping_erddap] running in process 1977264 2024-04-15 12:04:00,432 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:04:00,433 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-04-15 12:04:00,434 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:04:00,438 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2024-04-15 12:04:00,440 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-15 12:04:02,602 INFO [make_averaged_dataset] running in process 1977267 2024-04-15 12:04:02,604 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:04:02,605 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564 2024-04-15 12:04:02,606 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:04:02,608 INFO [make_averaged_dataset] creating day-averaged dataset for 15-Apr-2024 biology 2024-04-15 12:04:02,609 INFO [make_averaged_dataset] running in process 1977265 2024-04-15 12:04:02,611 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:04:02,612 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566 2024-04-15 12:04:02,613 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:04:02,614 INFO [make_averaged_dataset] creating day-averaged dataset for 15-Apr-2024 physics 2024-04-15 12:04:02,616 INFO [make_averaged_dataset] running in process 1977266 2024-04-15 12:04:02,617 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:04:02,618 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567 2024-04-15 12:04:02,619 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:04:02,621 INFO [make_averaged_dataset] creating day-averaged dataset for 15-Apr-2024 chemistry 2024-04-15 12:04:05,250 INFO [make_plots] running in process 1977262 2024-04-15 12:04:05,252 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:04:05,253 INFO [make_plots] publishing log messages to tcp://*:5569 2024-04-15 12:04:05,253 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:04:05,932 INFO [make_plots] preparing render list for 2024-04-15 NEMO nowcast-green research figures 2024-04-15 12:04:07,931 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:08,697 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:09,412 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:10,104 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:10,873 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:11,608 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:12,346 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:13,159 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:13,957 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:14,778 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:15,586 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:16,277 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:16,564 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:16,565 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:16,566 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:16,566 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-04-15 NEMO nowcast-green research render list 2024-04-15 12:04:16,567 INFO [make_plots] starting to render nemo nowcast-green research 15apr24 figures 2024-04-15 12:05:51,310 INFO [make_averaged_dataset] day-averaged dataset for 15-Apr-2024 physics created 2024-04-15 12:05:51,312 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-04-15 12:05:54,047 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 297, 45.8% complete 2024-04-15 12:06:10,750 INFO [make_averaged_dataset] day-averaged dataset for 15-Apr-2024 chemistry created 2024-04-15 12:06:10,755 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-04-15 12:06:38,123 INFO [make_averaged_dataset] day-averaged dataset for 15-Apr-2024 biology created 2024-04-15 12:06:38,126 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-04-15 12:08:49,434 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7611345 = 2024-04-15 18:29:20 UTC, 77.1% complete 2024-04-15 12:10:54,050 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 399, 61.6% complete 2024-04-15 12:13:49,895 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7611556 = 2024-04-15 20:50:00 UTC, 86.8% complete 2024-04-15 12:15:54,051 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete 2024-04-15 12:18:50,302 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7611765 = 2024-04-15 23:09:20 UTC, 96.5% complete 2024-04-15 12:20:54,051 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 603, 93.1% complete 2024-04-15 12:23:50,684 INFO [watch_NEMO_agrif] 15apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7611840 = 2024-04-15 23:59:20 UTC, 100.0% complete 2024-04-15 12:25:54,067 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-15 12:25:54,112 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-15 12:25:54,469 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-13'], host='localhost') 2024-04-15 12:25:55,965 INFO [download_wwatch3_results] running in process 1977860 2024-04-15 12:25:55,966 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:25:55,968 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564 2024-04-15 12:25:55,969 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:28:32,167 INFO [download_wwatch3_results] forecast 2024-04-13 results files from arbutus.cloud-nowcast downloaded 2024-04-15 12:28:32,171 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-15 12:28:32,356 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-13'], host='localhost') 2024-04-15 12:28:33,539 INFO [update_forecast_datasets] running in process 1977873 2024-04-15 12:28:33,596 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:28:33,635 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2024-04-15 12:28:33,636 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:28:33,637 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/13apr24 run 2024-04-15 12:28:33,639 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/13apr24 run 2024-04-15 12:28:33,667 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/13apr24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-04-15 12:28:33,667 INFO [update_forecast_datasets] wwatch3 2024-04-13 forecast rolling forecast datasets updated 2024-04-15 12:28:33,671 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-15 12:28:33,862 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-04-15 12:28:34,920 INFO [ping_erddap] running in process 1977878 2024-04-15 12:28:34,936 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:28:34,937 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-04-15 12:28:34,938 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:28:34,939 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-04-15 12:28:34,941 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-15 12:28:35,137 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-04-13'], host='localhost') 2024-04-15 12:28:40,849 INFO [make_plots] running in process 1977883 2024-04-15 12:28:40,850 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:28:40,851 INFO [make_plots] publishing log messages to tcp://*:5564 2024-04-15 12:28:40,852 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:28:40,853 INFO [make_plots] preparing render list for 2024-04-13 WaveWatch3 forecast publish figures 2024-04-15 12:28:40,854 INFO [make_plots] added Halibut Bank figure to 2024-04-13 WaveWatch3 forecast publish render list 2024-04-15 12:28:40,854 INFO [make_plots] added Sentry Shoal figure to 2024-04-13 WaveWatch3 forecast publish render list 2024-04-15 12:28:40,855 INFO [make_plots] starting to render wwatch3 forecast publish 13apr24 figures 2024-04-15 12:28:41,432 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-04-15 12:28:45,488 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-04-15 12:28:48,870 INFO [make_plots] finished rendering wwatch3 forecast publish 13apr24 figures 2024-04-15 12:28:48,871 INFO [make_plots] wwatch3 publish plots for 2024-04-13 forecast completed 2024-04-15 12:28:48,872 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 12:28:50,928 INFO [watch_NEMO_agrif] NEMO AGRIF run 11091780.orca2.ibb on orcinus-nowcast-agrif completed 2024-04-15 12:28:50,934 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker 2024-04-15 12:28:51,305 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 12:28:52,635 INFO [download_results] running in process 1977912 2024-04-15 12:28:52,636 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:28:52,638 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-15 12:28:52,639 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:28:52,640 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/15apr24 to /results/SalishSea/nowcast-agrif.201702 2024-04-15 12:29:31,924 INFO [download_results] nowcast-agrif 2024-04-15 results files from orcinus-nowcast-agrif downloaded 2024-04-15 12:29:31,931 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 12:29:32,122 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 12:29:36,689 INFO [make_plots] running in process 1977920 2024-04-15 12:29:36,690 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:29:36,691 INFO [make_plots] publishing log messages to tcp://*:5564 2024-04-15 12:29:36,692 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:29:37,319 INFO [make_plots] preparing render list for 2024-04-15 NEMO nowcast-agrif research figures 2024-04-15 12:29:37,320 INFO [make_plots] added baynes_sound_surface figure to 2024-04-15 NEMO nowcast-agrif research render list 2024-04-15 12:29:37,321 INFO [make_plots] starting to render nemo nowcast-agrif research 15apr24 figures 2024-04-15 12:29:46,805 INFO [make_plots] finished rendering nemo nowcast-agrif research 15apr24 figures 2024-04-15 12:29:46,806 INFO [make_plots] nemo research plots for 2024-04-15 nowcast-agrif completed 2024-04-15 12:29:46,807 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 12:30:22,774 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-04-15 12:31:15,631 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-04-15 12:31:15,864 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-14'], host='arbutus.cloud-nowcast') 2024-04-15 12:31:17,780 INFO [run_ww3] running in process 14384 2024-04-15 12:31:17,800 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:31:17,818 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-04-15 12:31:17,836 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:31:17,842 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-15T123117.779694-0700 2024-04-15 12:31:17,844 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-15T123117.779694-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-04-15 12:31:17,846 INFO [run_ww3] nowcast WaveWatch3 run for 2024-04-14 on arbutus.cloud-nowcast started 2024-04-15 12:31:17,848 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-15 12:31:18,014 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-04-15 12:31:19,858 INFO [watch_ww3] running in process 14478 2024-04-15 12:31:19,878 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:31:19,896 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-04-15 12:31:19,906 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:31:19,909 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-15 12:36:19,896 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 84, 19.4% complete 2024-04-15 12:41:19,902 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 183, 42.4% complete 2024-04-15 12:46:19,914 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 282, 65.3% complete 2024-04-15 12:51:19,911 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 381, 88.2% complete 2024-04-15 12:56:19,906 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-15 12:56:19,909 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-15 12:56:20,413 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-14'], host='localhost') 2024-04-15 12:56:20,419 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-14'], host='arbutus.cloud-nowcast') 2024-04-15 12:56:21,625 INFO [download_wwatch3_results] running in process 1978173 2024-04-15 12:56:21,626 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:56:21,627 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564 2024-04-15 12:56:21,628 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:56:22,369 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-15 12:56:22,458 INFO [run_ww3] running in process 18278 2024-04-15 12:56:22,459 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 12:56:22,460 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-04-15 12:56:22,462 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 12:56:22,465 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-15T125622.332047-0700 2024-04-15 12:56:22,513 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-15T125622.332047-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-04-15 12:56:22,515 INFO [run_ww3] forecast WaveWatch3 run for 2024-04-14 on arbutus.cloud-nowcast started 2024-04-15 12:56:22,576 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-04-15 12:56:25,047 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-15 12:58:26,875 INFO [download_wwatch3_results] nowcast 2024-04-14 results files from arbutus.cloud-nowcast downloaded 2024-04-15 12:58:26,878 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-15 13:01:24,956 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete 2024-04-15 13:06:24,996 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 189, 29.2% complete 2024-04-15 13:11:24,962 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 291, 44.9% complete 2024-04-15 13:16:25,055 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 396, 61.1% complete 2024-04-15 13:21:25,074 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 495, 76.4% complete 2024-04-15 13:23:42,696 INFO [make_plots] finished rendering nemo nowcast-green research 15apr24 figures 2024-04-15 13:23:42,706 INFO [make_plots] nemo research plots for 2024-04-15 nowcast-green completed 2024-04-15 13:23:42,768 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 13:26:24,979 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 594, 91.7% complete 2024-04-15 13:26:48,633 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 13:26:49,055 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-21'], host='localhost') 2024-04-15 13:26:49,061 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 13:26:49,064 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 13:26:50,424 INFO [download_results] running in process 1978253 2024-04-15 13:26:50,425 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 13:26:50,426 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-15 13:26:50,427 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 13:26:50,428 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21sep18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 13:26:55,584 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 13:27:56,136 INFO [download_results] hindcast 2018-09-21 results files from optimum-hindcast downloaded 2024-04-15 13:27:56,150 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 13:27:56,365 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-21'], host='localhost') 2024-04-15 13:27:57,640 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 13:31:24,970 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-15 13:31:25,073 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-15 13:31:25,373 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-14'], host='localhost') 2024-04-15 13:31:26,402 INFO [download_wwatch3_results] running in process 1978305 2024-04-15 13:31:26,403 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 13:31:26,404 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564 2024-04-15 13:31:26,405 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 13:34:09,039 INFO [download_wwatch3_results] forecast 2024-04-14 results files from arbutus.cloud-nowcast downloaded 2024-04-15 13:34:09,042 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-15 13:34:09,287 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-14'], host='localhost') 2024-04-15 13:34:10,449 INFO [update_forecast_datasets] running in process 1978322 2024-04-15 13:34:10,450 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 13:34:10,451 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564 2024-04-15 13:34:10,452 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-15 13:34:10,453 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/14apr24 run 2024-04-15 13:34:10,455 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/14apr24 run 2024-04-15 13:34:10,467 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/14apr24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-04-15 13:34:10,468 INFO [update_forecast_datasets] wwatch3 2024-04-14 forecast rolling forecast datasets updated 2024-04-15 13:34:10,471 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-15 13:34:10,687 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-04-15 13:34:11,744 INFO [ping_erddap] running in process 1978327 2024-04-15 13:34:11,745 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 13:34:11,746 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-04-15 13:34:11,747 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-15 13:34:11,748 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-04-15 13:34:11,751 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-15 13:34:11,965 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-04-14'], host='localhost') 2024-04-15 13:34:18,342 INFO [make_plots] running in process 1978332 2024-04-15 13:34:18,343 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 13:34:18,344 INFO [make_plots] publishing log messages to tcp://*:5564 2024-04-15 13:34:18,345 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 13:34:18,346 INFO [make_plots] preparing render list for 2024-04-14 WaveWatch3 forecast publish figures 2024-04-15 13:34:18,347 INFO [make_plots] added Halibut Bank figure to 2024-04-14 WaveWatch3 forecast publish render list 2024-04-15 13:34:18,348 INFO [make_plots] added Sentry Shoal figure to 2024-04-14 WaveWatch3 forecast publish render list 2024-04-15 13:34:18,349 INFO [make_plots] starting to render wwatch3 forecast publish 14apr24 figures 2024-04-15 13:34:18,860 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-04-15 13:34:28,544 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-04-15 13:34:35,097 INFO [make_plots] finished rendering wwatch3 forecast publish 14apr24 figures 2024-04-15 13:34:35,116 INFO [make_plots] wwatch3 publish plots for 2024-04-14 forecast completed 2024-04-15 13:34:35,151 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 14:02:48,638 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-04-15 14:06:27,312 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-04-15 14:06:27,522 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 14:06:30,485 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-15 14:06:30,707 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-04-15 14:06:33,104 INFO [watch_ww3] running in process 29845 2024-04-15 14:06:33,106 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 14:06:33,107 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-04-15 14:06:33,108 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 14:06:33,139 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-15 14:11:33,250 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 84, 19.4% complete 2024-04-15 14:16:33,245 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete 2024-04-15 14:21:33,284 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 291, 67.4% complete 2024-04-15 14:26:33,163 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 390, 90.3% complete 2024-04-15 14:31:33,153 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-15 14:31:33,153 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-15 14:31:33,600 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 14:31:33,609 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-15'], host='arbutus.cloud-nowcast') 2024-04-15 14:31:35,272 INFO [download_wwatch3_results] running in process 1978766 2024-04-15 14:31:35,273 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 14:31:35,275 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564 2024-04-15 14:31:35,277 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 14:31:36,431 INFO [run_ww3] running in process 1765 2024-04-15 14:31:36,451 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 14:31:36,452 INFO [run_ww3] publishing log messages to tcp://*:5572 2024-04-15 14:31:36,453 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 14:31:36,456 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-15T143136.435501-0700 2024-04-15 14:31:36,457 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-15T143136.435501-0700/SoGWW3.sh on arbutus.cloud-nowcast 2024-04-15 14:31:36,459 INFO [run_ww3] forecast WaveWatch3 run for 2024-04-15 on arbutus.cloud-nowcast started 2024-04-15 14:31:36,461 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-15 14:31:36,700 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-04-15 14:31:38,955 INFO [watch_ww3] running in process 1862 2024-04-15 14:31:38,957 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 14:31:38,958 INFO [watch_ww3] publishing log messages to tcp://*:5573 2024-04-15 14:31:38,959 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555 2024-04-15 14:31:39,200 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-15 14:34:35,495 INFO [download_wwatch3_results] nowcast 2024-04-15 results files from arbutus.cloud-nowcast downloaded 2024-04-15 14:34:35,499 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-15 14:36:39,199 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete 2024-04-15 14:41:39,201 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 201, 31.0% complete 2024-04-15 14:46:39,200 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete 2024-04-15 14:51:39,202 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 423, 65.3% complete 2024-04-15 14:51:59,278 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 14:51:59,703 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-26'], host='localhost') 2024-04-15 14:51:59,708 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 14:51:59,713 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 14:52:01,016 INFO [download_results] running in process 1978859 2024-04-15 14:52:01,018 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 14:52:01,018 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-15 14:52:01,019 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 14:52:01,020 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26sep18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 14:52:05,048 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 14:53:02,541 INFO [download_results] hindcast 2018-09-26 results files from optimum-hindcast downloaded 2024-04-15 14:53:02,554 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 14:53:02,769 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-26'], host='localhost') 2024-04-15 14:53:03,799 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 14:53:04,019 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-sep', 'robot.graham'], host='localhost') 2024-04-15 14:53:04,943 INFO [archive_tarball] running in process 1978909 2024-04-15 14:53:04,944 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 14:53:04,945 INFO [archive_tarball] publishing log messages to tcp://*:5564 2024-04-15 14:53:04,946 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-04-15 14:53:04,947 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-sep18.tar from /ocean/sallen/timbucktwo/oxygen/*sep18/ 2024-04-15 14:56:39,220 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 531, 81.9% complete 2024-04-15 15:00:42,182 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-sep18.index from /ocean/dlatorne/oxygen-sep18.tar 2024-04-15 15:00:42,289 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-sep18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-04-15 15:01:39,240 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 639, 98.6% complete 2024-04-15 15:06:39,210 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-15 15:06:39,213 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-15 15:06:39,619 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 15:06:40,840 INFO [download_wwatch3_results] running in process 1979019 2024-04-15 15:06:40,845 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 15:06:40,846 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566 2024-04-15 15:06:40,847 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 15:09:18,249 INFO [download_wwatch3_results] forecast 2024-04-15 results files from arbutus.cloud-nowcast downloaded 2024-04-15 15:09:18,252 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-15 15:09:18,468 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 15:09:19,435 INFO [update_forecast_datasets] running in process 1979041 2024-04-15 15:09:19,436 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 15:09:19,437 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566 2024-04-15 15:09:19,437 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-15 15:09:19,438 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/15apr24 run 2024-04-15 15:09:19,439 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/15apr24 run 2024-04-15 15:09:19,514 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/15apr24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-04-15 15:09:19,515 INFO [update_forecast_datasets] wwatch3 2024-04-15 forecast rolling forecast datasets updated 2024-04-15 15:09:19,518 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-15 15:09:19,736 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-04-15 15:09:20,548 INFO [ping_erddap] running in process 1979047 2024-04-15 15:09:20,551 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 15:09:20,551 INFO [ping_erddap] publishing log messages to tcp://*:5566 2024-04-15 15:09:20,552 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-15 15:09:20,553 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-04-15 15:09:20,555 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-15 15:09:20,770 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-04-15'], host='localhost') 2024-04-15 15:09:30,513 INFO [make_plots] running in process 1979052 2024-04-15 15:09:30,514 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 15:09:30,515 INFO [make_plots] publishing log messages to tcp://*:5566 2024-04-15 15:09:30,516 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-15 15:09:30,517 INFO [make_plots] preparing render list for 2024-04-15 WaveWatch3 forecast publish figures 2024-04-15 15:09:30,518 INFO [make_plots] added Halibut Bank figure to 2024-04-15 WaveWatch3 forecast publish render list 2024-04-15 15:09:30,519 INFO [make_plots] added Sentry Shoal figure to 2024-04-15 WaveWatch3 forecast publish render list 2024-04-15 15:09:30,520 INFO [make_plots] starting to render wwatch3 forecast publish 15apr24 figures 2024-04-15 15:09:31,107 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-04-15 15:09:40,306 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-04-15 15:09:47,470 INFO [make_plots] finished rendering wwatch3 forecast publish 15apr24 figures 2024-04-15 15:09:47,471 INFO [make_plots] wwatch3 publish plots for 2024-04-15 forecast completed 2024-04-15 15:09:47,474 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-15 15:10:22,847 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240415/18/ 2024-04-15 15:10:22,848 INFO [collect_weather] 2024-04-15 2.5km weather forecast 18 collection complete 2024-04-15 15:10:22,851 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-04-15 15:10:23,298 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2024-04-15 15:10:23,313 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2024-04-15 15:10:23,317 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2024-04-15 15:10:23,321 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-04-16'], host='localhost') 2024-04-15 15:10:24,491 INFO [download_weather] running in process 1979090 2024-04-15 15:10:24,492 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 15:10:24,494 INFO [download_weather] publishing log messages to tcp://*:5563 2024-04-15 15:10:24,495 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-15 15:10:24,496 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240415 2024-04-15 15:10:24,516 INFO [download_weather] running in process 1979092 2024-04-15 15:10:24,517 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 15:10:24,518 INFO [download_weather] publishing log messages to tcp://*:5566 2024-04-15 15:10:24,519 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-15 15:10:24,520 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240415 2024-04-15 15:10:24,529 INFO [collect_weather] running in process 1979093 2024-04-15 15:10:24,530 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 15:10:24,531 INFO [collect_weather] publishing log messages to tcp://*:5567 2024-04-15 15:10:24,531 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-15 15:10:24,601 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2024-04-15 15:10:25,788 INFO [crop_gribs] running in process 1979094 2024-04-15 15:10:25,789 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 15:10:25,790 INFO [crop_gribs] publishing log messages to tcp://*:5568 2024-04-15 15:10:25,791 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-04-15 15:10:25,792 INFO [crop_gribs] cropping 2024-04-16 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2024-04-15 15:10:25,803 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240416/00/ 2024-04-15 15:14:55,480 INFO [archive_tarball] hindcast *sep18 results files archived to robot.graham 2024-04-15 15:14:55,484 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-04-15 15:17:17,782 INFO [download_weather] 2024-04-15 1km weather forecast 12 downloads complete 2024-04-15 15:17:17,792 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-04-15 15:17:21,004 INFO [download_weather] 2024-04-15 1km weather forecast 00 downloads complete 2024-04-15 15:17:21,007 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-04-15 16:17:09,593 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 16:17:10,160 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-01'], host='localhost') 2024-04-15 16:17:10,188 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 16:17:10,193 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 16:17:11,505 INFO [download_results] running in process 1979387 2024-04-15 16:17:11,506 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 16:17:11,507 INFO [download_results] publishing log messages to tcp://*:5565 2024-04-15 16:17:11,508 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 16:17:11,509 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01oct18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 16:17:15,605 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 16:18:39,640 INFO [download_results] hindcast 2018-10-01 results files from optimum-hindcast downloaded 2024-04-15 16:18:39,655 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 16:18:39,870 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-01'], host='localhost') 2024-04-15 16:18:41,001 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 17:02:59,910 INFO [crop_gribs] crop_gribs 18 has watched for 8h; retrying remaining 1 file(s) 2024-04-15 17:03:06,171 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240415/18/ 2024-04-15 17:03:06,187 INFO [crop_gribs] 2024-04-15 18 GRIBs cropping complete 2024-04-15 17:03:06,209 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-04-15 17:47:20,405 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 17:47:20,842 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-06'], host='localhost') 2024-04-15 17:47:20,849 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 17:47:20,854 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 17:47:22,168 INFO [download_results] running in process 1979853 2024-04-15 17:47:22,169 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 17:47:22,170 INFO [download_results] publishing log messages to tcp://*:5562 2024-04-15 17:47:22,171 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 17:47:22,172 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06oct18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 17:47:26,313 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 17:48:29,326 INFO [download_results] hindcast 2018-10-06 results files from optimum-hindcast downloaded 2024-04-15 17:48:29,340 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 17:48:29,562 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-06'], host='localhost') 2024-04-15 17:48:30,552 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 19:12:30,823 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 19:12:31,225 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-11'], host='localhost') 2024-04-15 19:12:31,230 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 19:12:31,236 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 19:12:32,706 INFO [download_results] running in process 1980288 2024-04-15 19:12:32,707 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 19:12:32,708 INFO [download_results] publishing log messages to tcp://*:5563 2024-04-15 19:12:32,709 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 19:12:32,711 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11oct18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 19:12:36,697 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 19:13:39,590 INFO [download_results] hindcast 2018-10-11 results files from optimum-hindcast downloaded 2024-04-15 19:13:39,604 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 19:13:39,822 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-11'], host='localhost') 2024-04-15 19:13:41,010 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 20:37:41,545 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 20:37:42,059 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-16'], host='localhost') 2024-04-15 20:37:42,065 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 20:37:42,069 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 20:37:43,751 INFO [download_results] running in process 1981403 2024-04-15 20:37:43,752 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 20:37:43,753 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-15 20:37:43,754 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 20:37:43,755 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16oct18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 20:37:47,812 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 20:38:47,045 INFO [download_results] hindcast 2018-10-16 results files from optimum-hindcast downloaded 2024-04-15 20:38:47,057 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 20:38:47,271 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-16'], host='localhost') 2024-04-15 20:38:48,759 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 21:09:48,215 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240416/00/ 2024-04-15 21:09:48,216 INFO [collect_weather] 2024-04-16 2.5km weather forecast 00 collection complete 2024-04-15 21:09:48,219 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-04-15 21:09:48,609 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2024-04-15 21:09:48,614 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-04-16'], host='localhost') 2024-04-15 21:09:50,204 INFO [collect_weather] running in process 1981562 2024-04-15 21:09:50,205 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 21:09:50,207 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-04-15 21:09:50,208 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-15 21:09:50,222 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2024-04-15 21:09:52,134 INFO [crop_gribs] running in process 1981564 2024-04-15 21:09:52,135 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 21:09:52,135 INFO [crop_gribs] publishing log messages to tcp://*:5564 2024-04-15 21:09:52,136 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-04-15 21:09:52,137 INFO [crop_gribs] cropping 2024-04-16 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2024-04-15 21:09:52,148 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240416/06/ 2024-04-15 21:13:25,815 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240416/00/ 2024-04-15 21:13:25,815 INFO [crop_gribs] 2024-04-16 00 GRIBs cropping complete 2024-04-15 21:13:25,818 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-04-15 22:02:52,563 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 22:02:52,980 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-21'], host='localhost') 2024-04-15 22:02:52,986 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 22:02:52,994 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 22:02:54,685 INFO [download_results] running in process 1981767 2024-04-15 22:02:54,686 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 22:02:54,687 INFO [download_results] publishing log messages to tcp://*:5566 2024-04-15 22:02:54,688 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 22:02:54,689 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21oct18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 22:02:58,968 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 22:03:57,517 INFO [download_results] hindcast 2018-10-21 results files from optimum-hindcast downloaded 2024-04-15 22:03:57,534 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 22:03:57,755 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-21'], host='localhost') 2024-04-15 22:03:58,856 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 23:48:05,379 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-15 23:48:05,790 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-26'], host='localhost') 2024-04-15 23:48:05,795 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 23:48:05,799 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-15 23:48:06,884 INFO [download_results] running in process 1982742 2024-04-15 23:48:06,885 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 23:48:06,886 INFO [download_results] publishing log messages to tcp://*:5565 2024-04-15 23:48:06,887 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-15 23:48:06,888 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26oct18 to /ocean/sallen/timbucktwo/oxygen 2024-04-15 23:48:10,952 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-15 23:49:02,764 INFO [download_results] hindcast 2018-10-26 results files from optimum-hindcast downloaded 2024-04-15 23:49:02,777 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-15 23:49:02,990 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-26'], host='localhost') 2024-04-15 23:49:03,910 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-15 23:49:04,081 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-oct', 'robot.graham'], host='localhost') 2024-04-15 23:49:04,899 INFO [archive_tarball] running in process 1982789 2024-04-15 23:49:04,900 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-15 23:49:04,901 INFO [archive_tarball] publishing log messages to tcp://*:5563 2024-04-15 23:49:04,902 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-04-15 23:49:04,903 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct18.tar from /ocean/sallen/timbucktwo/oxygen/*oct18/ 2024-04-15 23:55:52,552 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct18.index from /ocean/dlatorne/oxygen-oct18.tar 2024-04-15 23:55:52,666 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-oct18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-04-16 00:24:29,674 INFO [archive_tarball] hindcast *oct18 results files archived to robot.graham 2024-04-16 00:24:29,677 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-04-16 01:13:15,585 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-16 01:13:15,991 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-01'], host='localhost') 2024-04-16 01:13:15,997 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-16 01:13:16,002 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-16 01:13:16,973 INFO [download_results] running in process 1983187 2024-04-16 01:13:16,975 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 01:13:16,976 INFO [download_results] publishing log messages to tcp://*:5565 2024-04-16 01:13:16,977 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-16 01:13:16,978 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01nov18 to /ocean/sallen/timbucktwo/oxygen 2024-04-16 01:13:21,167 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-16 01:14:05,396 INFO [download_results] hindcast 2018-11-01 results files from optimum-hindcast downloaded 2024-04-16 01:14:05,408 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-16 01:14:05,614 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-01'], host='localhost') 2024-04-16 01:14:06,471 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-16 02:38:25,562 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-16 02:38:25,962 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-06'], host='localhost') 2024-04-16 02:38:25,968 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-16 02:38:25,974 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-16 02:38:26,888 INFO [download_results] running in process 1984314 2024-04-16 02:38:26,889 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 02:38:26,890 INFO [download_results] publishing log messages to tcp://*:5565 2024-04-16 02:38:26,891 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-16 02:38:26,891 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06nov18 to /ocean/sallen/timbucktwo/oxygen 2024-04-16 02:38:31,060 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-16 02:39:15,633 INFO [download_results] hindcast 2018-11-06 results files from optimum-hindcast downloaded 2024-04-16 02:39:15,647 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-16 02:39:15,854 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-06'], host='localhost') 2024-04-16 02:39:16,927 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-16 03:01:07,791 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240416/06/ 2024-04-16 03:01:07,792 INFO [collect_weather] 2024-04-16 2.5km weather forecast 06 collection complete 2024-04-16 03:01:07,794 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-04-16 03:01:08,191 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,197 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,203 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,207 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,211 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,214 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,224 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,229 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,232 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,242 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,249 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,262 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,266 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,279 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-04-15'], host='localhost') 2024-04-16 03:01:08,292 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2024-04-16 03:01:08,317 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2024-04-16 03:01:08,328 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2024-04-16 03:01:08,352 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2024-04-16 03:01:08,370 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2024-04-16 03:01:08,387 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2024-04-16 03:01:10,298 INFO [collect_weather] running in process 1984521 2024-04-16 03:01:10,299 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:10,299 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-04-16 03:01:10,300 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:10,306 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2024-04-16 03:01:10,308 INFO [collect_NeahBay_ssh] running in process 1984520 2024-04-16 03:01:10,309 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:10,311 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5565 2024-04-16 03:01:10,312 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:10,313 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-04-16 2024-04-16 03:01:11,768 INFO [collect_river_data] running in process 1984512 2024-04-16 03:01:11,770 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,771 INFO [collect_river_data] publishing log messages to tcp://*:5566 2024-04-16 03:01:11,773 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,775 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-04-15 2024-04-16 03:01:11,810 INFO [collect_river_data] running in process 1984506 2024-04-16 03:01:11,813 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,815 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-04-16 03:01:11,816 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,818 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-04-15 2024-04-16 03:01:11,829 INFO [collect_river_data] running in process 1984508 2024-04-16 03:01:11,830 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,831 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-04-16 03:01:11,832 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,833 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-04-15 2024-04-16 03:01:11,872 INFO [collect_river_data] running in process 1984504 2024-04-16 03:01:11,873 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,874 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-04-16 03:01:11,875 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,876 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-04-15 2024-04-16 03:01:11,882 INFO [collect_river_data] running in process 1984502 2024-04-16 03:01:11,883 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,884 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-04-16 03:01:11,885 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,887 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-04-15 2024-04-16 03:01:11,899 INFO [collect_river_data] running in process 1984510 2024-04-16 03:01:11,900 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,901 INFO [collect_river_data] publishing log messages to tcp://*:5571 2024-04-16 03:01:11,902 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,902 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-04-15 2024-04-16 03:01:11,907 INFO [collect_river_data] running in process 1984513 2024-04-16 03:01:11,908 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,910 INFO [collect_river_data] publishing log messages to tcp://*:5572 2024-04-16 03:01:11,910 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,912 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-04-15 2024-04-16 03:01:11,922 INFO [collect_river_data] running in process 1984515 2024-04-16 03:01:11,924 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,925 INFO [collect_river_data] publishing log messages to tcp://*:5573 2024-04-16 03:01:11,926 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,926 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-04-15 2024-04-16 03:01:11,930 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2024-04-16 03:01:11,931 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2024-04-16 03:01:11,932 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:11,933 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-04-15 completed 2024-04-16 03:01:11,934 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-04-15 completed 2024-04-16 03:01:11,935 INFO [collect_river_data] running in process 1984516 2024-04-16 03:01:11,936 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,937 INFO [collect_river_data] publishing log messages to tcp://*:5574 2024-04-16 03:01:11,938 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,939 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-04-15 2024-04-16 03:01:11,946 INFO [collect_river_data] running in process 1984509 2024-04-16 03:01:11,951 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,952 INFO [collect_river_data] publishing log messages to tcp://*:5575 2024-04-16 03:01:11,953 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2024-04-16 03:01:11,953 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,954 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-04-15 completed 2024-04-16 03:01:11,955 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-04-15 2024-04-16 03:01:11,959 INFO [collect_river_data] running in process 1984507 2024-04-16 03:01:11,961 INFO [collect_river_data] running in process 1984514 2024-04-16 03:01:11,963 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,964 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,965 INFO [collect_river_data] publishing log messages to tcp://*:5576 2024-04-16 03:01:11,966 INFO [collect_river_data] publishing log messages to tcp://*:5577 2024-04-16 03:01:11,967 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,968 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,968 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-04-15 2024-04-16 03:01:11,969 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-04-15 2024-04-16 03:01:11,971 INFO [collect_river_data] running in process 1984511 2024-04-16 03:01:11,971 INFO [collect_river_data] running in process 1984505 2024-04-16 03:01:11,972 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,973 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:11,973 INFO [collect_river_data] publishing log messages to tcp://*:5579 2024-04-16 03:01:11,974 INFO [collect_river_data] publishing log messages to tcp://*:5578 2024-04-16 03:01:11,975 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,976 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:11,977 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-04-15 2024-04-16 03:01:11,979 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-04-15 2024-04-16 03:01:11,990 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/RobertsCreek_flow 2024-04-16 03:01:11,991 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-04-15 completed 2024-04-16 03:01:12,009 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2024-04-16 03:01:12,010 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-04-15 completed 2024-04-16 03:01:12,013 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-04-15 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2024-04-16 03:01:12,014 INFO [collect_river_data] ECCC Capilano river data collection for 2024-04-15 completed 2024-04-16 03:01:12,033 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2024-04-16 03:01:12,036 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-04-15 completed 2024-04-16 03:01:12,046 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2024-04-16 03:01:12,047 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-04-15 completed 2024-04-16 03:01:12,076 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-04-15 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2024-04-16 03:01:12,077 INFO [collect_river_data] ECCC Fraser river data collection for 2024-04-15 completed 2024-04-16 03:01:12,111 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2024-04-16 03:01:12,112 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-04-15 completed 2024-04-16 03:01:12,115 INFO [collect_NeahBay_ssh] 2024-04-16T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2024-04-16 03:01:12,116 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2024-04-16 03:01:12,117 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2024-04-16 03:01:12,118 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-04-15 completed 2024-04-16 03:01:12,120 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-04-15 completed 2024-04-16 03:01:12,121 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-04-15 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2024-04-16 03:01:12,122 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-04-15 completed 2024-04-16 03:01:12,127 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-04-15 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2024-04-16 03:01:12,128 INFO [collect_river_data] ECCC Englishman river data collection for 2024-04-15 completed 2024-04-16 03:01:12,229 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:12,488 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:12,646 INFO [crop_gribs] running in process 1984522 2024-04-16 03:01:12,647 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:12,648 INFO [crop_gribs] publishing log messages to tcp://*:5566 2024-04-16 03:01:12,648 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:12,649 INFO [crop_gribs] cropping 2024-04-16 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2024-04-16 03:01:12,661 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240416/12/ 2024-04-16 03:01:12,711 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:12,910 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:13,045 INFO [get_onc_ctd] running in process 1984518 2024-04-16 03:01:13,046 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:13,046 INFO [get_onc_ctd] publishing log messages to tcp://*:5568 2024-04-16 03:01:13,047 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:13,048 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-04-15 2024-04-16 03:01:13,054 INFO [get_onc_ctd] running in process 1984517 2024-04-16 03:01:13,054 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:13,055 INFO [get_onc_ctd] publishing log messages to tcp://*:5569 2024-04-16 03:01:13,056 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:13,057 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-04-15 2024-04-16 03:01:13,074 INFO [get_onc_ferry] running in process 1984519 2024-04-16 03:01:13,075 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:13,076 INFO [get_onc_ferry] publishing log messages to tcp://*:5590 2024-04-16 03:01:13,077 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:13,078 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-04-15 2024-04-16 03:01:13,098 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:13,285 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:13,469 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:13,667 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:13,846 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:14,029 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:14,210 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:14,390 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-04-16 03:01:14,605 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-04-16'], host='localhost') 2024-04-16 03:01:14,612 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:14,894 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-04-16 03:01:17,381 INFO [make_ssh_files] running in process 1984928 2024-04-16 03:01:17,382 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:17,383 INFO [make_ssh_files] publishing log messages to tcp://*:5565 2024-04-16 03:01:17,384 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:17,384 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-04-16 NOAA Neah Bay 00Z observation and forecast values 2024-04-16 03:01:22,252 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d11.nc 2024-04-16 03:01:22,439 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d12.nc 2024-04-16 03:01:22,630 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d13.nc 2024-04-16 03:01:22,815 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d14.nc 2024-04-16 03:01:23,006 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d15.nc 2024-04-16 03:01:23,315 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d16.nc 2024-04-16 03:01:23,491 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc 2024-04-16 03:01:23,700 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d18.nc 2024-04-16 03:01:23,872 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d19.nc 2024-04-16 03:01:24,074 INFO [make_ssh_files] sea surface height boundary file for 2024-04-16 forecast2 run created 2024-04-16 03:01:24,078 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-04-16 03:01:24,288 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-04-16 03:01:24,293 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-04-16 03:01:25,350 INFO [make_runoff_file] running in process 1984998 2024-04-16 03:01:25,351 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:25,352 INFO [make_runoff_file] publishing log messages to tcp://*:5565 2024-04-16 03:01:25,353 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:25,644 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-04-16 03:01:25,772 INFO [make_runoff_file] 2024-04-16 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-04-16 03:01:25,775 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-04-16 03:01:25,999 INFO [make_v202111_runoff_file] running in process 1984996 2024-04-16 03:01:26,000 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:26,001 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5567 2024-04-16 03:01:26,001 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:26,002 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-15 2024-04-16 03:01:27,147 ERROR [make_v202111_runoff_file] no 2024-04-15 discharge obs for Skagit_MountVernon: patching 2024-04-16 03:01:27,744 ERROR [make_v202111_runoff_file] no 2024-04-15 discharge obs for Snohomish_Monroe: patching 2024-04-16 03:01:27,969 ERROR [make_v202111_runoff_file] no 2024-04-15 discharge obs for Nisqually_McKenna: patching 2024-04-16 03:01:28,070 ERROR [make_v202111_runoff_file] no 2024-04-15 discharge obs for Greenwater_Greenwater: patching 2024-04-16 03:01:29,442 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-15: /results/forcing/rivers/R202108Dailies_y2024m04d15.nc 2024-04-16 03:01:29,443 INFO [make_v202111_runoff_file] 2024-04-15 runoff file creation completed 2024-04-16 03:01:29,444 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-04-16 03:01:31,596 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-04-15 2024-04-16 03:01:32,587 WARNING [get_onc_ctd] no SEVIP salinity data 2024-04-16 03:01:32,821 INFO [get_onc_ctd] 2024-04-15 ONC SEVIP CTD T&S file created 2024-04-16 03:01:32,823 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2024-04-16 03:01:33,043 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2024-04-16 03:01:33,806 INFO [ping_erddap] running in process 1985038 2024-04-16 03:01:33,807 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:01:33,808 INFO [ping_erddap] publishing log messages to tcp://*:5565 2024-04-16 03:01:33,809 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:01:33,811 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2024-04-16 03:01:33,812 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-16 03:01:48,648 CRITICAL [get_onc_ctd] 2024-04-15 ONC SCVIP CTD T&S file creation failed 2024-04-16 03:01:57,227 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-04-15 2024-04-16 03:02:24,561 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-04-15 2024-04-16 03:02:56,039 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-04-15 2024-04-16 03:03:12,779 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-04-15 2024-04-16 03:03:30,341 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-04-15 2024-04-16 03:03:38,848 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-04-15 2024-04-16 03:03:47,704 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-04-15; substituting empty dataset 2024-04-16 03:03:47,705 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-04-15 2024-04-16 03:03:55,725 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-15; substituting empty dataset 2024-04-16 03:04:54,281 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-04-15 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-04-16 03:04:54,299 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-15 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-04-16 03:04:54,577 INFO [get_onc_ferry] 2024-04-15 ONC TWDP ferry data file created 2024-04-16 03:04:54,580 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker 2024-04-16 03:04:54,757 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2024-04-16 03:04:55,570 INFO [ping_erddap] running in process 1985070 2024-04-16 03:04:55,571 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:04:55,572 INFO [ping_erddap] publishing log messages to tcp://*:5565 2024-04-16 03:04:55,574 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:04:55,575 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2024-04-16 03:04:55,577 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-16 03:09:55,947 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240416/06/ 2024-04-16 03:09:55,947 INFO [crop_gribs] 2024-04-16 06 GRIBs cropping complete 2024-04-16 03:09:55,950 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-04-16 03:09:56,358 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2024-04-16 03:09:57,886 INFO [grib_to_netcdf] running in process 1985089 2024-04-16 03:09:57,887 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:09:57,888 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564 2024-04-16 03:09:57,889 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:09:57,890 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-16 forecast2 run 2024-04-16 03:10:09,220 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d17.nc 2024-04-16 03:10:12,650 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d18.nc 2024-04-16 03:10:12,661 INFO [grib_to_netcdf] 2024-04-16 NEMO-atmos forcing file for forecast2 created 2024-04-16 03:10:12,663 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-04-16 03:10:12,877 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2024-04-16 03:10:12,880 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2024-04-16 03:10:12,884 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2024-04-16 03:10:12,890 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2024-04-16 03:10:13,766 INFO [upload_forcing] running in process 1985127 2024-04-16 03:10:13,767 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:10:13,768 INFO [upload_forcing] publishing log messages to tcp://*:5564 2024-04-16 03:10:13,769 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:10:13,770 INFO [upload_forcing] running in process 1985131 2024-04-16 03:10:13,772 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:10:13,773 INFO [upload_forcing] publishing log messages to tcp://*:5565 2024-04-16 03:10:13,774 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:10:13,781 INFO [upload_forcing] running in process 1985130 2024-04-16 03:10:13,783 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:10:13,784 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-04-16 03:10:13,784 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:10:13,815 INFO [upload_forcing] running in process 1985129 2024-04-16 03:10:13,816 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:10:13,817 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-04-16 03:10:13,818 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:10:15,155 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m04d15.nc 2024-04-16 03:10:15,208 INFO [upload_forcing] forecast2 2024-04-16 forcing files upload to optimum-hindcast completed 2024-04-16 03:10:15,214 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-16 03:10:15,340 INFO [upload_forcing] forecast2 2024-04-16 forcing files upload to orcinus-nowcast-agrif completed 2024-04-16 03:10:15,402 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-16 03:10:18,521 INFO [upload_forcing] forecast2 2024-04-16 forcing files upload to arbutus.cloud-nowcast completed 2024-04-16 03:10:18,526 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-16 03:10:18,744 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-16'], host='localhost') 2024-04-16 03:10:19,658 INFO [make_forcing_links] running in process 1985159 2024-04-16 03:10:19,659 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:10:19,660 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2024-04-16 03:10:19,661 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:10:21,146 INFO [make_forcing_links] forecast2 2024-04-16 forcing file links on arbutus.cloud-nowcast created 2024-04-16 03:10:21,151 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-04-16 03:10:21,373 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-16'], host='arbutus.cloud-nowcast') 2024-04-16 03:10:21,429 INFO [upload_forcing] forecast2 2024-04-16 forcing files upload to robot.graham completed 2024-04-16 03:10:21,434 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-04-16 03:10:23,337 INFO [run_NEMO] running in process 31843 2024-04-16 03:10:23,338 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:10:23,339 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-04-16 03:10:23,348 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:10:24,242 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/15apr24forecast2_2024-04-16T031023.418502-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-04-16 03:10:24,261 INFO [run_NEMO] forecast2 NEMO run for 2024-04-16 on arbutus.cloud-nowcast started 2024-04-16 03:10:24,266 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-04-16 03:10:24,487 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-04-16 03:10:26,595 INFO [watch_NEMO] running in process 32103 2024-04-16 03:10:26,596 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:10:26,597 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-04-16 03:10:26,598 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:10:26,666 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-04-16 03:15:26,668 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17593626 = 2024-04-17 04:43:20 UTC, 15.7% complete 2024-04-16 03:20:26,669 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17594089 = 2024-04-17 09:52:00 UTC, 32.9% complete 2024-04-16 03:25:26,670 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17594564 = 2024-04-17 15:08:40 UTC, 50.5% complete 2024-04-16 03:30:26,672 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17595027 = 2024-04-17 20:17:20 UTC, 67.7% complete 2024-04-16 03:35:26,673 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17595499 = 2024-04-18 01:32:00 UTC, 85.1% complete 2024-04-16 03:40:26,680 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17595900 = 2024-04-18 05:59:20 UTC, 100.0% complete 2024-04-16 03:45:26,682 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2024-04-16 03:45:26,686 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-04-16 03:45:27,107 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-16'], host='arbutus.cloud-nowcast') 2024-04-16 03:45:27,114 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-16'], host='arbutus.cloud-nowcast') 2024-04-16 03:45:27,118 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-15'], host='localhost') 2024-04-16 03:45:28,033 INFO [download_results] running in process 1985297 2024-04-16 03:45:28,034 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 03:45:28,034 INFO [download_results] publishing log messages to tcp://*:5564 2024-04-16 03:45:28,036 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-16 03:45:28,037 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/15apr24 to /results/SalishSea/forecast2.202111 2024-04-16 03:45:33,479 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-04-16 03:47:21,612 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-04-16 03:47:21,827 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-16'], host='arbutus.cloud-nowcast') 2024-04-16 03:47:33,469 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-04-16 03:47:33,690 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-04-16 03:47:46,060 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-04-16 03:52:46,060 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 99, 18.3% complete 2024-04-16 03:57:46,062 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 210, 38.9% complete 2024-04-16 04:02:46,075 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 318, 58.9% complete 2024-04-16 04:03:35,448 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-04-16 04:03:35,858 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-11'], host='localhost') 2024-04-16 04:03:35,864 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-16 04:03:35,868 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-04-16 04:03:36,806 INFO [download_results] running in process 1985468 2024-04-16 04:03:36,807 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:03:36,808 INFO [download_results] publishing log messages to tcp://*:5567 2024-04-16 04:03:36,809 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:03:36,810 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11nov18 to /ocean/sallen/timbucktwo/oxygen 2024-04-16 04:03:41,006 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-04-16 04:04:26,491 INFO [download_results] hindcast 2018-11-11 results files from optimum-hindcast downloaded 2024-04-16 04:04:26,505 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-16 04:04:26,718 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-11'], host='localhost') 2024-04-16 04:04:27,588 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-04-16 04:07:46,061 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 417, 77.2% complete 2024-04-16 04:10:16,342 INFO [download_results] forecast2 2024-04-15 results files from arbutus.cloud-nowcast downloaded 2024-04-16 04:10:16,348 INFO [manager] checklist updated with [results files] items from download_results worker 2024-04-16 04:10:16,564 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-04-15'], host='localhost') 2024-04-16 04:10:18,180 INFO [make_CHS_currents_file] running in process 1985575 2024-04-16 04:10:18,181 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:10:18,182 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563 2024-04-16 04:10:18,183 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:10:52,748 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-15 for forecast2 2024-04-16 04:10:52,751 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-04-16 04:10:52,970 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-04-15'], host='localhost') 2024-04-16 04:10:53,715 INFO [update_forecast_datasets] running in process 1985604 2024-04-16 04:10:53,716 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:10:53,717 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563 2024-04-16 04:10:53,718 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:10:53,719 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/15apr24 run 2024-04-16 04:12:45,393 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 525, 97.2% complete 2024-04-16 04:15:08,086 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/15apr24 run: /results/SalishSea/rolling-forecasts/nemo 2024-04-16 04:15:08,087 INFO [update_forecast_datasets] nemo 2024-04-15 forecast2 rolling forecast datasets updated 2024-04-16 04:15:08,089 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-16 04:15:08,305 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-04-16 04:15:08,312 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-04-15'], host='localhost') 2024-04-16 04:15:08,316 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-04-15'], host='localhost') 2024-04-16 04:15:09,262 INFO [ping_erddap] running in process 1985677 2024-04-16 04:15:09,263 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:15:09,264 INFO [ping_erddap] publishing log messages to tcp://*:5563 2024-04-16 04:15:09,265 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:15:09,273 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-04-16 04:15:09,274 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-16 04:15:10,371 INFO [make_surface_current_tiles] running in process 1985680 2024-04-16 04:15:10,373 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:15:10,375 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563 2024-04-16 04:15:10,377 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:15:10,402 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-15 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/15apr24 2024-04-16 04:15:13,316 INFO [make_plots] running in process 1985679 2024-04-16 04:15:13,317 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:15:13,318 INFO [make_plots] publishing log messages to tcp://*:5564 2024-04-16 04:15:13,319 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:15:14,242 INFO [make_plots] preparing render list for 2024-04-15 NEMO forecast2 publish figures 2024-04-16 04:15:14,414 INFO [make_plots] added Website_thumbnail figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,415 INFO [make_plots] added Threshold_website figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,416 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,418 INFO [make_plots] added Boundary Bay figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,418 INFO [make_plots] added Campbell River figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,419 INFO [make_plots] added Cherry Point figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,419 INFO [make_plots] added Friday Harbor figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,420 INFO [make_plots] added Halfmoon Bay figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,421 INFO [make_plots] added Nanaimo figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,422 INFO [make_plots] added Neah Bay figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,423 INFO [make_plots] added New Westminster figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,423 INFO [make_plots] added Patricia Bay figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,424 INFO [make_plots] added Point Atkinson figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,425 INFO [make_plots] added Port Renfrew figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,426 INFO [make_plots] added Sand Heads figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,426 INFO [make_plots] added Sandy Cove figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,427 INFO [make_plots] added Squamish figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,428 INFO [make_plots] added Victoria figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,429 INFO [make_plots] added Woodwards Landing figure to 2024-04-15 NEMO forecast2 publish render list 2024-04-16 04:15:14,430 INFO [make_plots] starting to render nemo forecast2 publish 15apr24 figures 2024-04-16 04:17:45,393 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2024-04-16 04:17:45,402 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-04-16 04:17:46,124 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-16'], host='localhost') 2024-04-16 04:17:47,468 INFO [download_wwatch3_results] running in process 1986205 2024-04-16 04:17:47,469 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:17:47,470 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567 2024-04-16 04:17:47,473 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:20:04,991 INFO [download_wwatch3_results] forecast2 2024-04-16 results files from arbutus.cloud-nowcast downloaded 2024-04-16 04:20:04,994 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-04-16 04:20:05,477 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-04-16'], host='localhost') 2024-04-16 04:20:06,932 INFO [update_forecast_datasets] running in process 1986530 2024-04-16 04:20:06,933 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:20:06,935 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567 2024-04-16 04:20:06,936 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:20:06,937 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/16apr24 run 2024-04-16 04:20:06,951 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/16apr24 run 2024-04-16 04:20:49,226 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/16apr24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-04-16 04:20:49,226 INFO [update_forecast_datasets] wwatch3 2024-04-16 forecast2 rolling forecast datasets updated 2024-04-16 04:20:49,230 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-04-16 04:20:49,664 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-04-16 04:20:50,865 INFO [ping_erddap] running in process 1986667 2024-04-16 04:20:50,866 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:20:50,866 INFO [ping_erddap] publishing log messages to tcp://*:5567 2024-04-16 04:20:50,868 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:20:50,869 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-04-16 04:20:50,871 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-04-16 04:20:51,317 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-04-16'], host='localhost') 2024-04-16 04:20:56,144 INFO [make_plots] running in process 1986672 2024-04-16 04:20:56,145 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:20:56,146 INFO [make_plots] publishing log messages to tcp://*:5567 2024-04-16 04:20:56,146 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:20:56,148 INFO [make_plots] preparing render list for 2024-04-16 WaveWatch3 forecast2 publish figures 2024-04-16 04:20:56,149 INFO [make_plots] added Halibut Bank figure to 2024-04-16 WaveWatch3 forecast2 publish render list 2024-04-16 04:20:56,149 INFO [make_plots] added Sentry Shoal figure to 2024-04-16 WaveWatch3 forecast2 publish render list 2024-04-16 04:20:56,150 INFO [make_plots] starting to render wwatch3 forecast2 publish 16apr24 figures 2024-04-16 04:20:56,507 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-04-16 04:21:07,166 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-04-16 04:21:16,623 INFO [make_plots] finished rendering wwatch3 forecast2 publish 16apr24 figures 2024-04-16 04:21:16,624 INFO [make_plots] wwatch3 publish plots for 2024-04-16 forecast2 completed 2024-04-16 04:21:16,627 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-16 04:29:55,604 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/15apr24 2024-04-16 04:33:30,598 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-04-16 04:33:30,599 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-04-16 04:33:32,328 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/15apr24 2024-04-16 04:33:32,337 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-15 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/15apr24 2024-04-16 04:33:32,338 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-15 forecast2 completed 2024-04-16 04:33:32,644 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-04-16 04:34:05,814 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:35:53,064 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:35:53,482 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:36:28,968 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:37:41,094 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:38:16,885 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:38:53,942 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:39:29,907 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:40:05,443 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:40:41,125 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:41:18,080 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:41:18,496 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:41:54,747 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:42:30,352 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-11 00:00:00Z to 2024-04-18 05:55:00Z 2024-04-16 04:43:05,926 INFO [make_plots] finished rendering nemo forecast2 publish 15apr24 figures 2024-04-16 04:43:05,930 INFO [make_plots] nemo publish plots for 2024-04-15 forecast2 completed 2024-04-16 04:43:05,939 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-04-16 04:43:06,275 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-04-15'], host='localhost') 2024-04-16 04:43:08,594 INFO [make_feeds] running in process 1988100 2024-04-16 04:43:08,595 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:43:08,596 INFO [make_feeds] publishing log messages to tcp://*:5563 2024-04-16 04:43:08,597 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:43:37,189 INFO [make_feeds] ATOM feeds for 2024-04-15 forecast2 run completed 2024-04-16 04:43:37,193 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-04-16 04:43:37,483 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2024-04-16 04:43:38,274 INFO [clear_checklist] running in process 1988124 2024-04-16 04:43:38,275 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:43:38,276 INFO [clear_checklist] publishing log messages to tcp://*:5563 2024-04-16 04:43:38,277 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:43:38,278 INFO [clear_checklist] requesting that manager clear system state checklist 2024-04-16 04:43:38,280 INFO [manager] writing checklist to log file 2024-04-16 04:43:38,325 INFO [manager] checklist cleared 2024-04-16 04:43:38,327 INFO [clear_checklist] nowcast system checklist cleared 2024-04-16 04:43:38,329 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2024-04-16 04:43:39,035 INFO [rotate_logs] running in process 1988129 2024-04-16 04:43:39,036 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-04-16 04:43:39,037 INFO [rotate_logs] publishing log messages to tcp://*:5563 2024-04-16 04:43:39,038 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2024-04-16 04:43:39,039 INFO [rotate_logs] rotating log files