2024-05-03 04:42:46,727 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-05-03 04:42:46,733 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-05-03 04:42:46,736 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-05-03 04:42:46,737 INFO [rotate_logs] log files rotated
2024-05-03 04:42:46,742 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-05-03 05:57:02,488 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 05:57:02,692 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-05-06'], host='localhost')
2024-05-03 05:57:02,698 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 05:57:02,704 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 05:57:03,605 INFO [download_results] running in process 2278875
2024-05-03 05:57:03,606 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 05:57:03,607 INFO [download_results] publishing log messages to tcp://*:5563
2024-05-03 05:57:03,608 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 05:57:03,609 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06may19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 05:57:07,807 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 05:57:44,284 INFO [download_results] hindcast 2019-05-06 results files from optimum-hindcast downloaded
2024-05-03 05:57:44,293 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 05:57:44,298 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-05-06'], host='localhost')
2024-05-03 05:57:45,078 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 07:22:11,913 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 07:22:12,110 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-05-11'], host='localhost')
2024-05-03 07:22:12,116 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 07:22:12,121 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 07:22:13,040 INFO [download_results] running in process 2280168
2024-05-03 07:22:13,041 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 07:22:13,042 INFO [download_results] publishing log messages to tcp://*:5563
2024-05-03 07:22:13,043 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 07:22:13,044 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11may19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 07:22:17,326 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 07:22:52,965 INFO [download_results] hindcast 2019-05-11 results files from optimum-hindcast downloaded
2024-05-03 07:22:52,977 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 07:22:52,982 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-05-11'], host='localhost')
2024-05-03 07:22:53,776 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 08:47:21,729 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 08:47:21,920 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-05-16'], host='localhost')
2024-05-03 08:47:21,926 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 08:47:21,930 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 08:47:23,062 INFO [download_results] running in process 2280857
2024-05-03 08:47:23,063 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 08:47:23,064 INFO [download_results] publishing log messages to tcp://*:5563
2024-05-03 08:47:23,065 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 08:47:23,066 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16may19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 08:47:27,336 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 08:48:17,242 INFO [download_results] hindcast 2019-05-16 results files from optimum-hindcast downloaded
2024-05-03 08:48:17,254 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 08:48:17,259 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-05-16'], host='localhost')
2024-05-03 08:48:18,247 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 09:03:32,739 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240503/12/
2024-05-03 09:03:32,740 INFO [collect_weather] 2024-05-03 2.5km weather forecast 12 collection complete
2024-05-03 09:03:32,745 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-05-03 09:03:32,952 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-05-02'], host='localhost')
2024-05-03 09:03:32,957 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-05-02'], host='localhost')
2024-05-03 09:03:32,961 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-05-02'], host='localhost')
2024-05-03 09:03:32,965 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-05-02'], host='localhost')
2024-05-03 09:03:32,970 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-05-03 09:03:32,974 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-05-03 09:03:32,980 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-05-03 09:03:32,984 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-05-03 09:03:32,989 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-05-03 09:03:34,276 INFO [collect_weather] running in process 2281745
2024-05-03 09:03:34,277 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:34,278 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-05-03 09:03:34,279 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:34,280 INFO [collect_NeahBay_ssh] running in process 2281743
2024-05-03 09:03:34,282 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:34,283 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-05-03 09:03:34,284 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:34,285 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-05-03
2024-05-03 09:03:34,294 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-05-03 09:03:34,501 INFO [download_live_ocean] running in process 2281744
2024-05-03 09:03:34,502 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:34,503 INFO [download_live_ocean] publishing log messages to tcp://*:5565
2024-05-03 09:03:34,506 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:34,507 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-05-03
2024-05-03 09:03:34,961 INFO [make_turbidity_file] running in process 2281742
2024-05-03 09:03:34,963 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:34,964 INFO [make_turbidity_file] publishing log messages to tcp://*:5566
2024-05-03 09:03:34,965 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:34,965 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-05-03
2024-05-03 09:03:35,003 INFO [collect_river_data] running in process 2281739
2024-05-03 09:03:35,004 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:35,005 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-05-03 09:03:35,006 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:35,007 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-05-02
2024-05-03 09:03:35,044 INFO [collect_river_data] running in process 2281737
2024-05-03 09:03:35,045 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:35,046 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-05-03 09:03:35,048 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:35,048 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-05-02
2024-05-03 09:03:35,097 INFO [collect_river_data] running in process 2281740
2024-05-03 09:03:35,099 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:35,101 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-05-03 09:03:35,102 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:35,104 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-05-02
2024-05-03 09:03:35,107 INFO [collect_river_data] running in process 2281741
2024-05-03 09:03:35,109 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:35,111 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-05-03 09:03:35,112 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:35,113 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-05-02
2024-05-03 09:03:35,226 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-05-02&endDT=2024-05-02 "HTTP/1.1 200 OK"
2024-05-03 09:03:35,231 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-05-02 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-05-03 09:03:35,232 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-05-02 completed
2024-05-03 09:03:35,234 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-05-03 09:03:35,265 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-05-02&endDT=2024-05-02 "HTTP/1.1 200 OK"
2024-05-03 09:03:35,274 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-05-02 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-05-03 09:03:35,276 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-05-02 completed
2024-05-03 09:03:35,278 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-05-03 09:03:35,375 INFO [download_live_ocean] downloaded 10143104 bytes from https://liveocean.apl.uw.edu/output/f2024.05.03/ubc.nc to /results/forcing/LiveOcean/downloaded/20240503/low_passed_UBC.nc
2024-05-03 09:03:35,377 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-05-03 09:03:35,392 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-05-02&endDT=2024-05-02 "HTTP/1.1 200 OK"
2024-05-03 09:03:35,398 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-05-02 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-05-03 09:03:35,399 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-05-02 completed
2024-05-03 09:03:35,400 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-05-03 09:03:35,408 INFO [crop_gribs] running in process 2281746
2024-05-03 09:03:35,410 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:35,411 INFO [crop_gribs] publishing log messages to tcp://*:5571
2024-05-03 09:03:35,411 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:35,412 INFO [crop_gribs] cropping 2024-05-03 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-05-03 09:03:35,423 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240503/18/
2024-05-03 09:03:35,427 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-05-02&endDT=2024-05-02 "HTTP/1.1 200 OK"
2024-05-03 09:03:35,432 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-05-02 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-05-03 09:03:35,433 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-05-02 completed
2024-05-03 09:03:35,436 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-05-03 09:03:36,047 INFO [collect_NeahBay_ssh] 2024-05-03T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-05-03 09:03:36,050 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-05-03 09:03:36,060 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 09:03:36,411 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240503/low_passed_UBC.nc
2024-05-03 09:03:36,412 INFO [download_live_ocean] 2024-05-03 Live Ocean file for Salish Sea western boundary downloaded
2024-05-03 09:03:36,414 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-05-03 09:03:36,600 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-05-03'], host='localhost')
2024-05-03 09:03:37,855 INFO [make_turbidity_file] 2024-05-03 Fraser River turbidity file creation complete
2024-05-03 09:03:37,857 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-05-03 09:03:38,469 INFO [make_live_ocean_files] running in process 2281918
2024-05-03 09:03:38,470 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:38,471 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2024-05-03 09:03:38,473 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:38,474 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-05-03 Live Ocean run
2024-05-03 09:03:38,586 INFO [make_ssh_files] running in process 2281905
2024-05-03 09:03:38,587 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:38,588 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2024-05-03 09:03:38,589 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:38,590 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-05-03 NOAA Neah Bay 06Z observation and forecast values
2024-05-03 09:03:43,143 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d29.nc
2024-05-03 09:03:43,318 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d30.nc
2024-05-03 09:03:43,533 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m05d01.nc
2024-05-03 09:03:43,717 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m05d02.nc
2024-05-03 09:03:43,897 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m05d03.nc
2024-05-03 09:03:44,061 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m05d04.nc
2024-05-03 09:03:44,233 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m05d05.nc
2024-05-03 09:03:44,400 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m05d06.nc
2024-05-03 09:03:44,501 INFO [make_ssh_files] sea surface height boundary file for 2024-05-03 nowcast run created
2024-05-03 09:03:44,505 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-05-03 09:03:44,516 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-05-03 09:03:44,519 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-05-03 09:03:45,545 INFO [make_runoff_file] running in process 2281963
2024-05-03 09:03:45,546 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:45,547 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2024-05-03 09:03:45,548 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:45,673 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-05-03 09:03:45,783 INFO [make_runoff_file] 2024-05-03 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-05-03 09:03:45,786 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-05-03 09:03:46,195 INFO [make_v202111_runoff_file] running in process 2281961
2024-05-03 09:03:46,197 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:03:46,198 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5565
2024-05-03 09:03:46,199 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:03:46,200 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-05-02
2024-05-03 09:03:48,665 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-05-02: /results/forcing/rivers/R202108Dailies_y2024m05d02.nc
2024-05-03 09:03:48,667 INFO [make_v202111_runoff_file] 2024-05-02 runoff file creation completed
2024-05-03 09:03:48,669 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-05-03 09:04:21,805 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2024-05-03 09:04:21,806 INFO [make_live_ocean_files] 2024-05-03 Live Ocean western boundary conditions files created
2024-05-03 09:04:21,809 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-05-03 09:12:37,695 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240503/12/
2024-05-03 09:12:37,696 INFO [crop_gribs] 2024-05-03 12 GRIBs cropping complete
2024-05-03 09:12:37,699 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-05-03 09:12:37,913 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-05-03 09:12:39,933 INFO [grib_to_netcdf] running in process 2282015
2024-05-03 09:12:39,934 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:12:39,935 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2024-05-03 09:12:39,936 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:12:39,937 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-05-03 nowcast and forecast runs
2024-05-03 09:12:51,623 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m05d03.nc
2024-05-03 09:13:02,275 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m05d04.nc
2024-05-03 09:13:08,375 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m05d05.nc
2024-05-03 09:13:08,390 INFO [grib_to_netcdf] 2024-05-03 NEMO-atmos forcing file for nowcast+ created
2024-05-03 09:13:08,393 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-05-03 09:13:08,403 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-05-03 09:13:08,407 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-05-03 09:13:08,411 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-05-03 09:13:08,415 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-05-03 09:13:08,419 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-05-03 09:13:09,258 INFO [ping_erddap] running in process 2282057
2024-05-03 09:13:09,259 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:13:09,260 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-05-03 09:13:09,262 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:13:09,265 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-05-03 09:13:09,268 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-05-03 09:13:09,410 INFO [upload_forcing] running in process 2282054
2024-05-03 09:13:09,412 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:13:09,413 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-05-03 09:13:09,414 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:13:09,432 INFO [upload_forcing] running in process 2282052
2024-05-03 09:13:09,433 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:13:09,434 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-05-03 09:13:09,435 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:13:09,479 INFO [upload_forcing] running in process 2282056
2024-05-03 09:13:09,481 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:13:09,482 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-05-03 09:13:09,483 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:13:09,490 INFO [upload_forcing] running in process 2282055
2024-05-03 09:13:09,491 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:13:09,492 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-05-03 09:13:09,493 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:13:11,743 INFO [upload_forcing] nowcast+ 2024-05-03 forcing files upload to optimum-hindcast completed
2024-05-03 09:13:11,747 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-05-03 09:13:11,794 INFO [upload_forcing] nowcast+ 2024-05-03 forcing files upload to orcinus-nowcast-agrif completed
2024-05-03 09:13:11,798 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-05-03 09:13:18,901 INFO [upload_forcing] nowcast+ 2024-05-03 forcing files upload to robot.graham completed
2024-05-03 09:13:18,905 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-05-03 09:13:23,326 INFO [upload_forcing] nowcast+ 2024-05-03 forcing files upload to arbutus.cloud-nowcast completed
2024-05-03 09:13:23,332 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-05-03 09:13:23,345 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 09:13:24,363 INFO [make_forcing_links] running in process 2282091
2024-05-03 09:13:24,364 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:13:24,366 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-05-03 09:13:24,367 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:13:25,713 INFO [make_forcing_links] nowcast+ 2024-05-03 forcing file links on arbutus.cloud-nowcast created
2024-05-03 09:13:25,719 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-05-03 09:13:25,731 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-05-03'], host='arbutus.cloud-nowcast')
2024-05-03 09:13:28,007 INFO [run_NEMO] running in process 17376
2024-05-03 09:13:28,009 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:13:28,010 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-05-03 09:13:28,022 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:13:28,895 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/03may24nowcast_2024-05-03T091328.061406-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-05-03 09:13:28,916 INFO [run_NEMO] nowcast NEMO run for 2024-05-03 on arbutus.cloud-nowcast started
2024-05-03 09:13:28,921 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-05-03 09:13:28,935 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-05-03 09:13:30,961 INFO [watch_NEMO] running in process 17629
2024-05-03 09:13:30,963 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:13:30,964 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-05-03 09:13:30,966 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:13:31,033 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-05-03 09:18:31,034 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17628164 = 2024-05-03 04:28:40 UTC, 18.7% complete
2024-05-03 09:23:31,036 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17628645 = 2024-05-03 09:49:20 UTC, 40.9% complete
2024-05-03 09:28:31,024 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17629110 = 2024-05-03 14:59:20 UTC, 62.5% complete
2024-05-03 09:33:31,012 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17629535 = 2024-05-03 19:42:40 UTC, 82.2% complete
2024-05-03 09:38:31,005 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17629920 = 2024-05-03 23:59:20 UTC, 100.0% complete
2024-05-03 09:43:31,020 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-05-03 09:43:31,023 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-05-03 09:43:31,226 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 09:43:31,232 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 09:43:32,328 INFO [download_results] running in process 2282183
2024-05-03 09:43:32,329 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:43:32,330 INFO [download_results] publishing log messages to tcp://*:5564
2024-05-03 09:43:32,331 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:43:32,333 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/03may24 to /results/SalishSea/nowcast-blue.202111
2024-05-03 09:43:32,335 INFO [make_forcing_links] running in process 2282182
2024-05-03 09:43:32,336 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:43:32,337 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-05-03 09:43:32,339 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:43:33,324 INFO [make_forcing_links] ssh 2024-05-03 forcing file links on arbutus.cloud-nowcast created
2024-05-03 09:43:33,328 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-05-03 09:43:33,344 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-05-03'], host='arbutus.cloud-nowcast')
2024-05-03 09:43:35,435 INFO [run_NEMO] running in process 18921
2024-05-03 09:43:35,437 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:43:35,438 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-05-03 09:43:35,440 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:43:36,439 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-05-03 09:43:36,443 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/03may24forecast_2024-05-03T094335.587042-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-05-03 09:43:36,447 INFO [run_NEMO] forecast NEMO run for 2024-05-03 on arbutus.cloud-nowcast started
2024-05-03 09:43:36,455 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-05-03 09:43:38,432 INFO [watch_NEMO] running in process 19175
2024-05-03 09:43:38,433 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 09:43:38,435 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-05-03 09:43:38,436 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-05-03 09:43:38,503 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-05-03 09:48:38,532 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17630345 = 2024-05-04 04:42:40 UTC, 13.1% complete
2024-05-03 09:53:38,820 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17630816 = 2024-05-04 09:56:40 UTC, 27.6% complete
2024-05-03 09:58:38,912 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17631296 = 2024-05-04 15:16:40 UTC, 42.5% complete
2024-05-03 10:03:36,130 INFO [download_results] nowcast 2024-05-03 results files from arbutus.cloud-nowcast downloaded
2024-05-03 10:03:36,135 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 10:03:36,147 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:03:36,153 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-05-02'], host='localhost')
2024-05-03 10:03:36,157 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:03:38,013 INFO [make_CHS_currents_file] running in process 2282271
2024-05-03 10:03:38,014 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:03:38,015 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-05-03 10:03:38,016 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:03:38,494 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17631786 = 2024-05-04 20:43:20 UTC, 57.6% complete
2024-05-03 10:03:39,245 INFO [make_plots] running in process 2282268
2024-05-03 10:03:39,246 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:03:39,247 INFO [make_plots] publishing log messages to tcp://*:5565
2024-05-03 10:03:39,249 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:03:39,290 INFO [make_plots] running in process 2282270
2024-05-03 10:03:39,293 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:03:39,295 INFO [make_plots] publishing log messages to tcp://*:5566
2024-05-03 10:03:39,296 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:04:05,502 INFO [make_CHS_currents_file] Made CHS currents file for 2024-05-03 for nowcast
2024-05-03 10:04:05,506 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-05-03 10:04:20,225 INFO [make_plots] preparing render list for 2024-05-03 NEMO nowcast-blue research figures
2024-05-03 10:04:20,230 INFO [make_plots] preparing render list for 2024-05-02 NEMO nowcast-blue comparison figures
2024-05-03 10:04:21,020 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-blue research render list
2024-05-03 10:04:21,601 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-blue research render list
2024-05-03 10:04:21,602 INFO [make_plots] added Currents_sections_and_surface figure to 2024-05-03 NEMO nowcast-blue research render list
2024-05-03 10:04:21,603 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-05-03 NEMO nowcast-blue research render list
2024-05-03 10:04:21,604 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-05-03 NEMO nowcast-blue research render list
2024-05-03 10:04:21,605 INFO [make_plots] starting to render nemo nowcast research 03may24 figures
2024-05-03 10:05:34,761 INFO [make_plots] finished rendering nemo nowcast research 03may24 figures
2024-05-03 10:05:34,763 INFO [make_plots] nemo research plots for 2024-05-03 nowcast completed
2024-05-03 10:05:34,782 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-05-03 10:06:17,971 INFO [make_plots] added SH_wind figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,972 INFO [make_plots] added Compare_VENUS_East figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,973 INFO [make_plots] added Compare_VENUS_Central figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,975 INFO [make_plots] added Central_ADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,976 INFO [make_plots] added Central_depavADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,977 INFO [make_plots] added Central_timeavADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,978 INFO [make_plots] added East_ADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,979 INFO [make_plots] added East_depavADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,980 INFO [make_plots] added East_timeavADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,981 INFO [make_plots] added ddl_ADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,981 INFO [make_plots] added ddl_depavADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,982 INFO [make_plots] added ddl_timeavADCP figure to 2024-05-02 NEMO nowcast-blue comparison render list
2024-05-03 10:06:17,983 INFO [make_plots] starting to render nemo nowcast comparison 02may24 figures
2024-05-03 10:08:15,503 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-05-03 10:08:15,959 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-05-03 10:08:16,422 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-05-03 10:08:16,878 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-05-03 10:08:17,339 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-05-03 10:08:17,809 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-05-03 10:08:18,276 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-05-03 10:08:18,735 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-05-03 10:08:19,196 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-05-03 10:08:19,197 INFO [make_plots] finished rendering nemo nowcast comparison 02may24 figures
2024-05-03 10:08:19,279 INFO [make_plots] nemo comparison plots for 2024-05-02 nowcast completed
2024-05-03 10:08:19,283 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-05-03 10:08:38,492 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17632260 = 2024-05-05 01:59:20 UTC, 72.2% complete
2024-05-03 10:12:31,826 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 10:12:32,033 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-05-21'], host='localhost')
2024-05-03 10:12:32,041 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 10:12:32,047 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 10:12:33,153 INFO [download_results] running in process 2282391
2024-05-03 10:12:33,155 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:12:33,157 INFO [download_results] publishing log messages to tcp://*:5565
2024-05-03 10:12:33,159 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:12:33,162 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21may19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 10:12:37,302 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 10:13:16,555 INFO [download_results] hindcast 2019-05-21 results files from optimum-hindcast downloaded
2024-05-03 10:13:16,564 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 10:13:16,580 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-05-21'], host='localhost')
2024-05-03 10:13:17,604 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 10:13:38,491 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17632724 = 2024-05-05 07:08:40 UTC, 86.5% complete
2024-05-03 10:18:38,489 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17633160 = 2024-05-05 11:59:20 UTC, 100.0% complete
2024-05-03 10:23:38,492 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-05-03 10:23:38,496 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-05-03 10:23:38,716 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-05-03'], host='arbutus.cloud-nowcast')
2024-05-03 10:23:38,723 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-05-03'], host='arbutus.cloud-nowcast')
2024-05-03 10:23:38,727 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:23:38,732 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:23:38,739 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:23:38,743 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:23:38,747 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:23:39,770 INFO [upload_forcing] running in process 2282460
2024-05-03 10:23:39,771 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:23:39,773 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-05-03 10:23:39,774 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:23:39,776 INFO [upload_forcing] running in process 2282461
2024-05-03 10:23:39,777 INFO [upload_forcing] running in process 2282459
2024-05-03 10:23:39,779 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:23:39,781 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:23:39,782 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-05-03 10:23:39,783 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-05-03 10:23:39,785 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:23:39,786 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:23:39,788 INFO [upload_forcing] running in process 2282457
2024-05-03 10:23:39,790 INFO [download_results] running in process 2282462
2024-05-03 10:23:39,791 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:23:39,793 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:23:39,794 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-05-03 10:23:39,796 INFO [download_results] publishing log messages to tcp://*:5568
2024-05-03 10:23:39,797 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:23:39,799 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:23:39,801 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/03may24 to /results/SalishSea/forecast.202111
2024-05-03 10:23:40,448 INFO [upload_forcing] turbidity 2024-05-03 forcing files upload to optimum-hindcast completed
2024-05-03 10:23:40,453 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-05-03 10:23:40,601 INFO [upload_forcing] turbidity 2024-05-03 forcing files upload to orcinus-nowcast-agrif completed
2024-05-03 10:23:40,606 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-05-03 10:23:40,626 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:23:40,850 INFO [upload_forcing] turbidity 2024-05-03 forcing files upload to arbutus.cloud-nowcast completed
2024-05-03 10:23:40,855 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-05-03 10:23:40,876 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:23:41,622 INFO [make_forcing_links] running in process 2282493
2024-05-03 10:23:41,623 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:23:41,625 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-05-03 10:23:41,626 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:23:41,874 INFO [make_forcing_links] running in process 2282495
2024-05-03 10:23:41,876 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:23:41,877 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2024-05-03 10:23:41,877 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:23:42,459 INFO [make_forcing_links] nowcast-agrif 2024-05-03 forcing file links on orcinus-nowcast-agrif created
2024-05-03 10:23:42,463 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-05-03 10:23:42,481 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:23:43,093 INFO [make_forcing_links] nowcast-green 2024-05-03 forcing file links on arbutus.cloud-nowcast created
2024-05-03 10:23:43,097 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-05-03 10:23:43,114 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-05-03'], host='arbutus.cloud-nowcast')
2024-05-03 10:23:43,505 INFO [run_NEMO_agrif] running in process 2282509
2024-05-03 10:23:43,509 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:23:43,510 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5565
2024-05-03 10:23:43,511 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:23:43,692 INFO [upload_forcing] turbidity 2024-05-03 forcing files upload to robot.graham completed
2024-05-03 10:23:43,696 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-05-03 10:23:45,887 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-05-03 10:23:46,610 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/03may24nowcast-green_2024-05-03T102345.709948-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-05-03 10:23:46,628 INFO [run_NEMO] nowcast-green NEMO run for 2024-05-03 on arbutus.cloud-nowcast started
2024-05-03 10:23:46,633 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-05-03 10:23:46,651 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-05-03 10:23:49,739 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-05-03 10:23:59,327 INFO [run_NEMO_agrif] job id for 03may24nowcast-agrif: 11094769.orca2.ibb
2024-05-03 10:23:59,328 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-05-03 queued on orcinus-nowcast-agrif
2024-05-03 10:23:59,333 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-05-03 10:23:59,356 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11094769.orca2.ibb'], host='localhost')
2024-05-03 10:24:00,307 INFO [watch_NEMO_agrif] running in process 2282528
2024-05-03 10:24:00,308 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:24:00,309 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5562
2024-05-03 10:24:00,310 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:24:01,130 INFO [watch_NEMO_agrif] watching 03may24nowcast-agrif job 11094769 on orcinus-nowcast-agrif
2024-05-03 10:24:01,366 INFO [watch_NEMO_agrif] 03may24nowcast-agrif job 11094769 is queued on orcinus-nowcast-agrif
2024-05-03 10:24:24,959 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-05-03 10:24:24,982 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-05-03'], host='arbutus.cloud-nowcast')
2024-05-03 10:24:36,039 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-05-03 10:24:36,059 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-05-03 10:24:47,859 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-05-03 10:25:01,581 INFO [watch_NEMO_agrif] 03may24nowcast-agrif job 11094769 is queued on orcinus-nowcast-agrif
2024-05-03 10:26:02,638 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7648561 = 2024-05-03 00:00:00 UTC, 0.0% complete
2024-05-03 10:28:50,329 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17627918 = 2024-05-03 01:44:40 UTC, 7.3% complete
2024-05-03 10:29:47,873 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 84, 19.4% complete
2024-05-03 10:31:03,091 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7648771 = 2024-05-03 02:20:00 UTC, 9.7% complete
2024-05-03 10:33:50,355 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17628094 = 2024-05-03 03:42:00 UTC, 15.4% complete
2024-05-03 10:34:47,813 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 174, 40.3% complete
2024-05-03 10:36:03,573 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7648983 = 2024-05-03 04:41:20 UTC, 19.5% complete
2024-05-03 10:38:50,016 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17628272 = 2024-05-03 05:40:40 UTC, 23.7% complete
2024-05-03 10:39:47,642 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 264, 61.1% complete
2024-05-03 10:41:04,020 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7649195 = 2024-05-03 07:02:40 UTC, 29.4% complete
2024-05-03 10:43:50,073 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17628449 = 2024-05-03 07:38:40 UTC, 31.9% complete
2024-05-03 10:44:47,648 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 354, 81.9% complete
2024-05-03 10:46:04,469 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7649409 = 2024-05-03 09:25:20 UTC, 39.3% complete
2024-05-03 10:48:50,049 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17628626 = 2024-05-03 09:36:40 UTC, 40.1% complete
2024-05-03 10:49:47,604 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-05-03 10:51:04,954 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7649624 = 2024-05-03 11:48:40 UTC, 49.2% complete
2024-05-03 10:52:57,862 INFO [download_results] forecast 2024-05-03 results files from arbutus.cloud-nowcast downloaded
2024-05-03 10:52:57,867 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 10:52:57,887 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:52:59,765 INFO [make_CHS_currents_file] running in process 2282581
2024-05-03 10:52:59,766 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:52:59,767 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-05-03 10:52:59,768 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:53:40,116 INFO [make_CHS_currents_file] Made CHS currents file for 2024-05-03 for forecast
2024-05-03 10:53:40,119 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-05-03 10:53:40,142 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:53:40,920 INFO [update_forecast_datasets] running in process 2282605
2024-05-03 10:53:40,921 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:53:40,922 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-05-03 10:53:40,924 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:53:40,925 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/03may24 run
2024-05-03 10:53:41,186 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/03may24 run: /results/SalishSea/rolling-forecasts/nemo
2024-05-03 10:53:41,187 INFO [update_forecast_datasets] nemo 2024-05-03 forecast rolling forecast datasets updated
2024-05-03 10:53:41,189 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-05-03 10:53:41,208 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-05-03 10:53:41,212 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:53:41,216 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:53:42,132 INFO [ping_erddap] running in process 2282610
2024-05-03 10:53:42,134 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:53:42,135 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-05-03 10:53:42,136 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:53:42,146 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-05-03 10:53:42,148 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-05-03 10:53:43,197 INFO [make_surface_current_tiles] running in process 2282613
2024-05-03 10:53:43,199 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:53:43,200 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-05-03 10:53:43,201 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:53:43,213 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-05-03 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/03may24
2024-05-03 10:53:44,591 INFO [make_plots] running in process 2282612
2024-05-03 10:53:44,593 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:53:44,596 INFO [make_plots] publishing log messages to tcp://*:5566
2024-05-03 10:53:44,598 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:53:45,666 INFO [make_plots] preparing render list for 2024-05-03 NEMO forecast publish figures
2024-05-03 10:53:45,860 INFO [make_plots] added Website_thumbnail figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,861 INFO [make_plots] added Threshold_website figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,862 INFO [make_plots] added PA_tidal_predictions figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,863 INFO [make_plots] added Boundary Bay figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,864 INFO [make_plots] added Campbell River figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,865 INFO [make_plots] added Cherry Point figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,866 INFO [make_plots] added Friday Harbor figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,867 INFO [make_plots] added Halfmoon Bay figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,868 INFO [make_plots] added Nanaimo figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,869 INFO [make_plots] added Neah Bay figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,869 INFO [make_plots] added New Westminster figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,870 INFO [make_plots] added Patricia Bay figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,871 INFO [make_plots] added Point Atkinson figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,871 INFO [make_plots] added Port Renfrew figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,872 INFO [make_plots] added Sand Heads figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,873 INFO [make_plots] added Sandy Cove figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,874 INFO [make_plots] added Squamish figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,874 INFO [make_plots] added Victoria figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,875 INFO [make_plots] added Woodwards Landing figure to 2024-05-03 NEMO forecast publish render list
2024-05-03 10:53:45,876 INFO [make_plots] starting to render nemo forecast publish 03may24 figures
2024-05-03 10:53:49,746 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17628803 = 2024-05-03 11:34:40 UTC, 48.3% complete
2024-05-03 10:54:47,327 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-05-03 10:54:47,421 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-05-03 10:54:47,745 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 10:54:47,766 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-05-03'], host='arbutus.cloud-nowcast')
2024-05-03 10:54:49,407 INFO [download_wwatch3_results] running in process 2282891
2024-05-03 10:54:49,408 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:54:49,409 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2024-05-03 10:54:49,411 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:54:49,749 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-05-03 10:54:49,776 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-05-03 10:54:52,186 INFO [watch_ww3] running in process 23414
2024-05-03 10:54:52,187 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 10:54:52,190 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-05-03 10:54:52,191 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-05-03 10:54:52,223 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-05-03 10:56:05,439 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7649835 = 2024-05-03 14:09:20 UTC, 59.0% complete
2024-05-03 10:56:50,411 INFO [download_wwatch3_results] nowcast 2024-05-03 results files from arbutus.cloud-nowcast downloaded
2024-05-03 10:56:50,415 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-05-03 10:58:49,778 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17628980 = 2024-05-03 13:32:40 UTC, 56.5% complete
2024-05-03 10:59:52,238 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 84, 13.0% complete
2024-05-03 11:01:05,889 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7650049 = 2024-05-03 16:32:00 UTC, 68.9% complete
2024-05-03 11:03:49,807 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17629156 = 2024-05-03 15:30:00 UTC, 64.6% complete
2024-05-03 11:04:52,204 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 174, 26.9% complete
2024-05-03 11:06:06,370 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7650263 = 2024-05-03 18:54:40 UTC, 78.8% complete
2024-05-03 11:07:41,893 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/03may24
2024-05-03 11:08:49,739 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17629333 = 2024-05-03 17:28:00 UTC, 72.8% complete
2024-05-03 11:09:52,185 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 270, 41.7% complete
2024-05-03 11:11:06,818 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7650475 = 2024-05-03 21:16:00 UTC, 88.7% complete
2024-05-03 11:11:29,279 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-05-03 11:11:29,280 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-05-03 11:11:34,625 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/03may24
2024-05-03 11:11:34,635 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-05-03 forecast into /results/nowcast-sys/figures/surface_currents/forecast/03may24
2024-05-03 11:11:34,636 INFO [make_surface_current_tiles] surface current tile figures for 2024-05-03 forecast completed
2024-05-03 11:11:34,967 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-05-03 11:12:04,036 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:13:49,738 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17629510 = 2024-05-03 19:26:00 UTC, 81.0% complete
2024-05-03 11:13:52,355 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:13:52,769 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:14:27,803 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:14:52,185 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 354, 54.6% complete
2024-05-03 11:15:38,741 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:16:07,267 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7650689 = 2024-05-03 23:38:40 UTC, 98.6% complete
2024-05-03 11:16:15,249 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:16:51,393 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:17:27,702 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:18:03,232 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:18:40,161 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:18:49,756 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17629687 = 2024-05-03 21:24:00 UTC, 89.2% complete
2024-05-03 11:19:16,690 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:19:17,116 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:19:52,178 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 444, 68.5% complete
2024-05-03 11:19:52,759 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:20:29,367 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-28 00:00:00Z to 2024-05-05 11:55:00Z
2024-05-03 11:21:05,153 INFO [make_plots] finished rendering nemo forecast publish 03may24 figures
2024-05-03 11:21:05,156 INFO [make_plots] nemo publish plots for 2024-05-03 forecast completed
2024-05-03 11:21:05,164 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-05-03 11:21:05,325 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 11:21:07,699 INFO [watch_NEMO_agrif] 03may24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7650720 = 2024-05-03 23:59:20 UTC, 100.0% complete
2024-05-03 11:21:08,091 INFO [make_feeds] running in process 2284700
2024-05-03 11:21:08,092 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:21:08,094 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-05-03 11:21:08,096 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:21:34,983 INFO [make_feeds] ATOM feeds for 2024-05-03 forecast run completed
2024-05-03 11:21:34,986 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-05-03 11:23:49,732 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17629863 = 2024-05-03 23:21:20 UTC, 97.4% complete
2024-05-03 11:24:52,181 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 540, 83.3% complete
2024-05-03 11:26:07,938 INFO [watch_NEMO_agrif] NEMO AGRIF run 11094769.orca2.ibb on orcinus-nowcast-agrif completed
2024-05-03 11:26:07,942 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-05-03 11:26:08,320 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 11:26:09,439 INFO [download_results] running in process 2284731
2024-05-03 11:26:09,440 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:26:09,442 INFO [download_results] publishing log messages to tcp://*:5562
2024-05-03 11:26:09,443 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:26:09,444 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/03may24 to /results/SalishSea/nowcast-agrif.201702
2024-05-03 11:26:32,646 INFO [download_results] nowcast-agrif 2024-05-03 results files from orcinus-nowcast-agrif downloaded
2024-05-03 11:26:32,652 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 11:26:32,827 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 11:26:36,049 INFO [make_plots] running in process 2284739
2024-05-03 11:26:36,050 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:26:36,052 INFO [make_plots] publishing log messages to tcp://*:5562
2024-05-03 11:26:36,053 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:26:36,675 INFO [make_plots] preparing render list for 2024-05-03 NEMO nowcast-agrif research figures
2024-05-03 11:26:36,676 INFO [make_plots] added baynes_sound_surface figure to 2024-05-03 NEMO nowcast-agrif research render list
2024-05-03 11:26:36,677 INFO [make_plots] starting to render nemo nowcast-agrif research 03may24 figures
2024-05-03 11:26:45,170 INFO [make_plots] finished rendering nemo nowcast-agrif research 03may24 figures
2024-05-03 11:26:45,172 INFO [make_plots] nemo research plots for 2024-05-03 nowcast-agrif completed
2024-05-03 11:26:45,176 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-05-03 11:28:49,725 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17629920 = 2024-05-03 23:59:20 UTC, 100.0% complete
2024-05-03 11:29:52,172 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 633, 97.7% complete
2024-05-03 11:33:49,720 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-05-03 11:33:49,724 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-05-03 11:33:50,090 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 11:33:51,213 INFO [download_results] running in process 2284799
2024-05-03 11:33:51,215 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:33:51,216 INFO [download_results] publishing log messages to tcp://*:5562
2024-05-03 11:33:51,218 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:33:51,219 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/03may24 to /results2/SalishSea/nowcast-green.202111
2024-05-03 11:34:52,240 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-05-03 11:34:52,243 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-05-03 11:34:52,617 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 11:34:53,544 INFO [download_wwatch3_results] running in process 2284809
2024-05-03 11:34:53,545 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:34:53,547 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-05-03 11:34:53,548 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:40:33,867 INFO [download_wwatch3_results] forecast 2024-05-03 results files from arbutus.cloud-nowcast downloaded
2024-05-03 11:40:33,872 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-05-03 11:40:34,058 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 11:40:35,001 INFO [update_forecast_datasets] running in process 2284823
2024-05-03 11:40:35,002 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:40:35,004 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-05-03 11:40:35,005 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:40:35,006 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/03may24 run
2024-05-03 11:40:35,007 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/03may24 run
2024-05-03 11:40:35,128 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/03may24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-05-03 11:40:35,129 INFO [update_forecast_datasets] wwatch3 2024-05-03 forecast rolling forecast datasets updated
2024-05-03 11:40:35,132 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-05-03 11:40:35,320 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-05-03 11:40:36,115 INFO [ping_erddap] running in process 2284828
2024-05-03 11:40:36,116 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:40:36,118 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-05-03 11:40:36,119 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:40:36,121 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-05-03 11:40:36,125 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-05-03 11:40:36,301 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 11:40:39,380 INFO [make_plots] running in process 2284833
2024-05-03 11:40:39,381 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:40:39,382 INFO [make_plots] publishing log messages to tcp://*:5565
2024-05-03 11:40:39,383 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:40:39,384 INFO [make_plots] preparing render list for 2024-05-03 WaveWatch3 forecast publish figures
2024-05-03 11:40:39,385 INFO [make_plots] added Halibut Bank figure to 2024-05-03 WaveWatch3 forecast publish render list
2024-05-03 11:40:39,386 INFO [make_plots] added Sentry Shoal figure to 2024-05-03 WaveWatch3 forecast publish render list
2024-05-03 11:40:39,386 INFO [make_plots] starting to render wwatch3 forecast publish 03may24 figures
2024-05-03 11:40:39,573 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-05-03 11:40:48,136 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-05-03 11:40:55,416 INFO [make_plots] finished rendering wwatch3 forecast publish 03may24 figures
2024-05-03 11:40:55,417 INFO [make_plots] wwatch3 publish plots for 2024-05-03 forecast completed
2024-05-03 11:40:55,421 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-05-03 11:57:43,802 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 11:57:44,197 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-05-26'], host='localhost')
2024-05-03 11:57:44,203 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 11:57:44,207 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 11:57:45,231 INFO [download_results] running in process 2284892
2024-05-03 11:57:45,232 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:57:45,233 INFO [download_results] publishing log messages to tcp://*:5565
2024-05-03 11:57:45,234 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:57:45,236 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26may19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 11:57:49,386 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 11:58:33,027 INFO [download_results] hindcast 2019-05-26 results files from optimum-hindcast downloaded
2024-05-03 11:58:33,040 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 11:58:33,217 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-05-26'], host='localhost')
2024-05-03 11:58:34,185 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 11:58:34,437 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-may', 'robot.graham'], host='localhost')
2024-05-03 11:58:35,592 INFO [archive_tarball] running in process 2284935
2024-05-03 11:58:35,594 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 11:58:35,595 INFO [archive_tarball] publishing log messages to tcp://*:5564
2024-05-03 11:58:35,596 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-05-03 11:58:35,597 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-may19.tar from /ocean/sallen/timbucktwo/oxygen/*may19/
2024-05-03 12:04:31,510 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-may19.index from /ocean/dlatorne/oxygen-may19.tar
2024-05-03 12:04:31,596 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-may19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-05-03 12:10:35,549 INFO [download_results] nowcast-green 2024-05-03 results files from arbutus.cloud-nowcast downloaded
2024-05-03 12:10:35,555 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 12:10:35,740 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 12:10:35,746 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-05-03 12:10:35,750 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 12:10:35,755 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 12:10:35,759 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-05-03'], host='localhost')
2024-05-03 12:10:36,902 INFO [ping_erddap] running in process 2285010
2024-05-03 12:10:36,903 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 12:10:36,904 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-05-03 12:10:36,906 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-05-03 12:10:36,912 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-05-03 12:10:36,915 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-05-03 12:10:38,373 INFO [make_averaged_dataset] running in process 2285012
2024-05-03 12:10:38,374 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 12:10:38,375 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2024-05-03 12:10:38,376 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-05-03 12:10:38,377 INFO [make_averaged_dataset] creating day-averaged dataset for 03-May-2024 chemistry
2024-05-03 12:10:38,381 INFO [make_averaged_dataset] running in process 2285013
2024-05-03 12:10:38,382 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 12:10:38,383 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2024-05-03 12:10:38,384 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-05-03 12:10:38,385 INFO [make_averaged_dataset] creating day-averaged dataset for 03-May-2024 biology
2024-05-03 12:10:38,386 INFO [make_averaged_dataset] running in process 2285011
2024-05-03 12:10:38,387 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 12:10:38,388 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567
2024-05-03 12:10:38,390 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-05-03 12:10:38,391 INFO [make_averaged_dataset] creating day-averaged dataset for 03-May-2024 physics
2024-05-03 12:10:40,208 INFO [make_plots] running in process 2285008
2024-05-03 12:10:40,209 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 12:10:40,210 INFO [make_plots] publishing log messages to tcp://*:5568
2024-05-03 12:10:40,211 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-05-03 12:10:40,830 INFO [make_plots] preparing render list for 2024-05-03 NEMO nowcast-green research figures
2024-05-03 12:10:42,826 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:43,586 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:44,328 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:45,062 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:45,801 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:46,551 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:47,300 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:48,192 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:49,024 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:49,861 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:50,690 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:51,479 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:51,744 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:51,746 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:51,747 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:51,749 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-05-03 NEMO nowcast-green research render list
2024-05-03 12:10:51,749 INFO [make_plots] starting to render nemo nowcast-green research 03may24 figures
2024-05-03 12:12:00,633 INFO [make_averaged_dataset] day-averaged dataset for 03-May-2024 physics created
2024-05-03 12:12:00,637 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-05-03 12:12:57,167 INFO [make_averaged_dataset] day-averaged dataset for 03-May-2024 chemistry created
2024-05-03 12:12:57,170 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-05-03 12:13:26,728 INFO [make_averaged_dataset] day-averaged dataset for 03-May-2024 biology created
2024-05-03 12:13:26,732 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-05-03 12:33:28,756 INFO [archive_tarball] hindcast *may19 results files archived to robot.graham
2024-05-03 12:33:28,759 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-05-03 13:22:54,023 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 13:22:54,511 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-06-01'], host='localhost')
2024-05-03 13:22:54,516 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 13:22:54,521 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 13:22:56,546 INFO [download_results] running in process 2285763
2024-05-03 13:22:56,578 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 13:22:56,625 INFO [download_results] publishing log messages to tcp://*:5565
2024-05-03 13:22:56,658 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 13:22:56,660 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jun19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 13:23:00,827 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 13:23:40,908 INFO [download_results] hindcast 2019-06-01 results files from optimum-hindcast downloaded
2024-05-03 13:23:40,947 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 13:23:41,130 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-06-01'], host='localhost')
2024-05-03 13:23:42,106 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 13:40:05,556 INFO [make_plots] finished rendering nemo nowcast-green research 03may24 figures
2024-05-03 13:40:05,567 INFO [make_plots] nemo research plots for 2024-05-03 nowcast-green completed
2024-05-03 13:40:05,629 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-05-03 14:48:05,506 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 14:48:06,088 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-06-06'], host='localhost')
2024-05-03 14:48:06,115 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 14:48:06,121 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 14:48:07,799 INFO [download_results] running in process 2286208
2024-05-03 14:48:07,800 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 14:48:07,801 INFO [download_results] publishing log messages to tcp://*:5562
2024-05-03 14:48:07,802 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 14:48:07,803 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06jun19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 14:48:11,844 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 14:48:57,686 INFO [download_results] hindcast 2019-06-06 results files from optimum-hindcast downloaded
2024-05-03 14:48:57,696 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 14:48:57,926 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-06-06'], host='localhost')
2024-05-03 14:48:59,204 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 15:13:46,003 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240503/18/
2024-05-03 15:13:46,038 INFO [collect_weather] 2024-05-03 2.5km weather forecast 18 collection complete
2024-05-03 15:13:46,135 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-05-03 15:13:47,255 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-05-03 15:13:47,446 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-05-03 15:13:47,579 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-05-03 15:13:47,709 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-05-04'], host='localhost')
2024-05-03 15:14:08,386 INFO [collect_weather] running in process 2286694
2024-05-03 15:14:08,502 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 15:14:08,528 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-05-03 15:14:08,546 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-05-03 15:14:08,551 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-05-03 15:14:08,554 INFO [download_weather] running in process 2286691
2024-05-03 15:14:08,555 INFO [download_weather] running in process 2286693
2024-05-03 15:14:08,558 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 15:14:08,559 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 15:14:08,561 INFO [download_weather] publishing log messages to tcp://*:5563
2024-05-03 15:14:08,565 INFO [download_weather] publishing log messages to tcp://*:5564
2024-05-03 15:14:08,566 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-05-03 15:14:08,568 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-05-03 15:14:08,569 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240503
2024-05-03 15:14:08,571 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240503
2024-05-03 15:14:43,684 INFO [crop_gribs] running in process 2286695
2024-05-03 15:14:43,727 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 15:14:43,755 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-05-03 15:14:43,786 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-05-03 15:14:43,787 INFO [crop_gribs] cropping 2024-05-04 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-05-03 15:14:43,788 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240504/00/
2024-05-03 15:18:09,543 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240503/18/
2024-05-03 15:18:09,657 INFO [crop_gribs] 2024-05-03 18 GRIBs cropping complete
2024-05-03 15:18:09,766 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-05-03 15:21:59,977 INFO [download_weather] 2024-05-03 1km weather forecast 00 downloads complete
2024-05-03 15:21:59,981 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-05-03 15:22:52,057 INFO [download_weather] 2024-05-03 1km weather forecast 12 downloads complete
2024-05-03 15:22:52,085 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-05-03 16:13:16,588 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 16:13:17,545 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-06-11'], host='localhost')
2024-05-03 16:13:17,562 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 16:13:17,572 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 16:13:23,897 INFO [download_results] running in process 2287027
2024-05-03 16:13:23,912 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 16:13:23,913 INFO [download_results] publishing log messages to tcp://*:5563
2024-05-03 16:13:23,915 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 16:13:23,916 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11jun19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 16:13:28,223 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 16:14:24,956 INFO [download_results] hindcast 2019-06-11 results files from optimum-hindcast downloaded
2024-05-03 16:14:24,968 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 16:14:25,210 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-06-11'], host='localhost')
2024-05-03 16:14:28,813 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 17:38:33,108 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 17:38:33,684 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-06-16'], host='localhost')
2024-05-03 17:38:33,695 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 17:38:33,700 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 17:38:36,426 INFO [download_results] running in process 2287880
2024-05-03 17:38:36,427 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 17:38:36,428 INFO [download_results] publishing log messages to tcp://*:5564
2024-05-03 17:38:36,429 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 17:38:36,430 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16jun19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 17:38:40,440 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 17:39:26,190 INFO [download_results] hindcast 2019-06-16 results files from optimum-hindcast downloaded
2024-05-03 17:39:26,201 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 17:39:26,420 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-06-16'], host='localhost')
2024-05-03 17:39:27,723 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 19:03:45,124 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 19:03:45,519 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-06-21'], host='localhost')
2024-05-03 19:03:45,526 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 19:03:45,531 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 19:03:46,821 INFO [download_results] running in process 2288554
2024-05-03 19:03:46,822 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 19:03:46,823 INFO [download_results] publishing log messages to tcp://*:5564
2024-05-03 19:03:46,824 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 19:03:46,825 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21jun19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 19:03:50,867 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 19:04:26,537 INFO [download_results] hindcast 2019-06-21 results files from optimum-hindcast downloaded
2024-05-03 19:04:26,549 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 19:04:26,757 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-06-21'], host='localhost')
2024-05-03 19:04:27,692 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 20:28:55,052 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 20:28:55,454 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-06-26'], host='localhost')
2024-05-03 20:28:55,461 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 20:28:55,467 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 20:28:56,536 INFO [download_results] running in process 2288956
2024-05-03 20:28:56,537 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 20:28:56,538 INFO [download_results] publishing log messages to tcp://*:5563
2024-05-03 20:28:56,540 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 20:28:56,541 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jun19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 20:29:00,617 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 20:29:36,835 INFO [download_results] hindcast 2019-06-26 results files from optimum-hindcast downloaded
2024-05-03 20:29:36,845 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 20:29:37,052 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-06-26'], host='localhost')
2024-05-03 20:29:37,883 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 20:29:38,099 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-jun', 'robot.graham'], host='localhost')
2024-05-03 20:29:38,941 INFO [archive_tarball] running in process 2289006
2024-05-03 20:29:38,942 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 20:29:38,943 INFO [archive_tarball] publishing log messages to tcp://*:5563
2024-05-03 20:29:38,944 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-05-03 20:29:38,945 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jun19.tar from /ocean/sallen/timbucktwo/oxygen/*jun19/
2024-05-03 20:34:31,092 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jun19.index from /ocean/dlatorne/oxygen-jun19.tar
2024-05-03 20:34:31,470 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jun19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-05-03 21:00:06,796 INFO [archive_tarball] hindcast *jun19 results files archived to robot.graham
2024-05-03 21:00:06,799 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-05-03 21:02:42,711 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240504/00/
2024-05-03 21:02:42,712 INFO [collect_weather] 2024-05-04 2.5km weather forecast 00 collection complete
2024-05-03 21:02:42,715 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-05-03 21:02:43,104 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-05-03 21:02:43,110 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-05-04'], host='localhost')
2024-05-03 21:02:44,252 INFO [collect_weather] running in process 2289458
2024-05-03 21:02:44,252 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 21:02:44,254 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-05-03 21:02:44,255 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-05-03 21:02:44,268 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-05-03 21:02:46,359 INFO [crop_gribs] running in process 2289460
2024-05-03 21:02:46,360 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 21:02:46,361 INFO [crop_gribs] publishing log messages to tcp://*:5563
2024-05-03 21:02:46,363 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-05-03 21:02:46,364 INFO [crop_gribs] cropping 2024-05-04 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-05-03 21:02:46,371 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240504/06/
2024-05-03 21:12:41,875 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240504/00/
2024-05-03 21:12:41,876 INFO [crop_gribs] 2024-05-04 00 GRIBs cropping complete
2024-05-03 21:12:41,878 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-05-03 21:54:05,070 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 21:54:05,448 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-01'], host='localhost')
2024-05-03 21:54:05,454 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 21:54:05,460 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 21:54:06,389 INFO [download_results] running in process 2289723
2024-05-03 21:54:06,390 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 21:54:06,392 INFO [download_results] publishing log messages to tcp://*:5565
2024-05-03 21:54:06,393 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 21:54:06,395 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jul19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 21:54:10,632 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 21:54:46,330 INFO [download_results] hindcast 2019-07-01 results files from optimum-hindcast downloaded
2024-05-03 21:54:46,341 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 21:54:46,557 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-01'], host='localhost')
2024-05-03 21:54:47,337 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-03 23:24:15,190 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-03 23:24:15,603 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-06'], host='localhost')
2024-05-03 23:24:15,609 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 23:24:15,614 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-03 23:24:16,567 INFO [download_results] running in process 2290030
2024-05-03 23:24:16,569 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-03 23:24:16,571 INFO [download_results] publishing log messages to tcp://*:5566
2024-05-03 23:24:16,572 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-03 23:24:16,573 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06jul19 to /ocean/sallen/timbucktwo/oxygen
2024-05-03 23:24:20,586 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-03 23:24:56,483 INFO [download_results] hindcast 2019-07-06 results files from optimum-hindcast downloaded
2024-05-03 23:24:56,494 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-03 23:24:56,708 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-06'], host='localhost')
2024-05-03 23:24:57,530 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-04 00:49:25,188 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-04 00:49:25,603 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-11'], host='localhost')
2024-05-04 00:49:25,610 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-04 00:49:25,614 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-04 00:49:26,559 INFO [download_results] running in process 2290357
2024-05-04 00:49:26,560 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-04 00:49:26,562 INFO [download_results] publishing log messages to tcp://*:5566
2024-05-04 00:49:26,563 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-04 00:49:26,564 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11jul19 to /ocean/sallen/timbucktwo/oxygen
2024-05-04 00:49:30,734 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-04 00:50:06,470 INFO [download_results] hindcast 2019-07-11 results files from optimum-hindcast downloaded
2024-05-04 00:50:06,479 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-04 00:50:06,686 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-11'], host='localhost')
2024-05-04 00:50:07,503 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-05-04 02:14:35,264 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-05-04 02:14:35,670 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-16'], host='localhost')
2024-05-04 02:14:35,679 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-04 02:14:35,683 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-05-04 02:14:36,603 INFO [download_results] running in process 2290803
2024-05-04 02:14:36,604 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-05-04 02:14:36,605 INFO [download_results] publishing log messages to tcp://*:5565
2024-05-04 02:14:36,607 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-05-04 02:14:36,608 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16jul19 to /ocean/sallen/timbucktwo/oxygen
2024-05-04 02:14:40,604 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-05-04 02:15:16,545 INFO [download_results] hindcast 2019-07-16 results files from optimum-hindcast downloaded
2024-05-04 02:15:16,556 INFO [manager] checklist updated with [results files] items from download_results worker
2024-05-04 02:15:16,775 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-16'], host='localhost')
2024-05-04 02:15:17,651 INFO [manager] checklist updated with [results splitting] items from split_results worker