2024-07-20 04:32:47,884 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-07-20 04:32:47,893 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-07-20 04:32:47,897 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-07-20 04:32:47,899 INFO [rotate_logs] log files rotated
2024-07-20 04:32:47,904 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-07-20 05:24:40,137 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 05:24:40,289 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-03-11'], host='localhost')
2024-07-20 05:24:40,296 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 05:24:40,300 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 05:24:41,454 INFO [download_results] running in process 699418
2024-07-20 05:24:41,455 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 05:24:41,456 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-20 05:24:41,457 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 05:24:41,458 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11mar18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 05:24:45,847 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 05:25:21,286 INFO [download_results] hindcast 2018-03-11 results files from optimum-hindcast downloaded
2024-07-20 05:25:21,298 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 05:25:21,303 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-03-11'], host='localhost')
2024-07-20 05:25:22,326 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 06:49:50,024 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 06:49:50,171 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-03-16'], host='localhost')
2024-07-20 06:49:50,177 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 06:49:50,182 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 06:49:51,339 INFO [download_results] running in process 700026
2024-07-20 06:49:51,340 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 06:49:51,341 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-20 06:49:51,342 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 06:49:51,343 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16mar18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 06:49:55,550 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 06:50:31,212 INFO [download_results] hindcast 2018-03-16 results files from optimum-hindcast downloaded
2024-07-20 06:50:31,223 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 06:50:31,231 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-03-16'], host='localhost')
2024-07-20 06:50:32,297 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 08:14:59,979 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 08:15:00,148 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-03-21'], host='localhost')
2024-07-20 08:15:00,154 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 08:15:00,159 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 08:15:01,483 INFO [download_results] running in process 700317
2024-07-20 08:15:01,485 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 08:15:01,486 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-20 08:15:01,487 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 08:15:01,487 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21mar18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 08:15:05,725 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 08:15:41,627 INFO [download_results] hindcast 2018-03-21 results files from optimum-hindcast downloaded
2024-07-20 08:15:41,638 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 08:15:41,643 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-03-21'], host='localhost')
2024-07-20 08:15:42,640 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 09:03:17,708 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240720/12/
2024-07-20 09:03:17,709 INFO [collect_weather] 2024-07-20 2.5km weather forecast 12 collection complete
2024-07-20 09:03:17,712 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-20 09:03:17,901 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-07-19'], host='localhost')
2024-07-20 09:03:17,907 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-07-19'], host='localhost')
2024-07-20 09:03:17,912 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-07-19'], host='localhost')
2024-07-20 09:03:17,917 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-07-19'], host='localhost')
2024-07-20 09:03:17,923 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-07-20 09:03:17,927 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-07-20 09:03:17,931 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-07-20 09:03:17,938 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-07-20 09:03:17,942 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-07-20 09:03:19,264 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240720/12/
2024-07-20 09:03:19,265 INFO [crop_gribs] 2024-07-20 12 GRIBs cropping complete
2024-07-20 09:03:19,267 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-20 09:03:19,442 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-07-20 09:03:19,535 INFO [collect_weather] running in process 700705
2024-07-20 09:03:19,536 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:19,547 INFO [collect_NeahBay_ssh] running in process 700703
2024-07-20 09:03:19,548 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:19,549 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5565
2024-07-20 09:03:19,549 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:19,550 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-07-20
2024-07-20 09:03:19,551 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-07-20 09:03:19,552 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:19,559 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-07-20 09:03:19,791 INFO [download_live_ocean] running in process 700704
2024-07-20 09:03:19,793 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:19,794 INFO [download_live_ocean] publishing log messages to tcp://*:5566
2024-07-20 09:03:19,794 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:19,795 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-07-20
2024-07-20 09:03:20,141 INFO [collect_river_data] running in process 700696
2024-07-20 09:03:20,142 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:20,144 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-20 09:03:20,145 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:20,145 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-07-19
2024-07-20 09:03:20,153 INFO [collect_river_data] running in process 700700
2024-07-20 09:03:20,154 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:20,154 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-20 09:03:20,155 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:20,156 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-07-19
2024-07-20 09:03:20,192 INFO [collect_river_data] running in process 700698
2024-07-20 09:03:20,193 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:20,196 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-20 09:03:20,197 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:20,198 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-07-19
2024-07-20 09:03:20,233 INFO [make_turbidity_file] running in process 700701
2024-07-20 09:03:20,234 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:20,235 INFO [make_turbidity_file] publishing log messages to tcp://*:5570
2024-07-20 09:03:20,236 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:20,236 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-07-20
2024-07-20 09:03:20,237 INFO [collect_river_data] running in process 700699
2024-07-20 09:03:20,239 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:20,240 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-20 09:03:20,241 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:20,242 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-07-19
2024-07-20 09:03:20,354 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-07-19&endDT=2024-07-19 "HTTP/1.1 200 OK"
2024-07-20 09:03:20,361 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-07-20 09:03:20,362 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-07-19 completed
2024-07-20 09:03:20,364 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-20 09:03:20,487 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-07-19&endDT=2024-07-19 "HTTP/1.1 200 OK"
2024-07-20 09:03:20,492 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-07-20 09:03:20,493 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-07-19 completed
2024-07-20 09:03:20,494 INFO [crop_gribs] running in process 700706
2024-07-20 09:03:20,495 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-07-19&endDT=2024-07-19 "HTTP/1.1 200 OK"
2024-07-20 09:03:20,496 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:20,497 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-20 09:03:20,498 INFO [crop_gribs] publishing log messages to tcp://*:5572
2024-07-20 09:03:20,499 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:20,500 INFO [crop_gribs] cropping 2024-07-20 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-07-20 09:03:20,504 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-07-20 09:03:20,505 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-07-19 completed
2024-07-20 09:03:20,508 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-20 09:03:20,514 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240720/18/
2024-07-20 09:03:20,561 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-07-19&endDT=2024-07-19 "HTTP/1.1 200 OK"
2024-07-20 09:03:20,566 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-07-20 09:03:20,567 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-07-19 completed
2024-07-20 09:03:20,568 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-20 09:03:20,778 INFO [download_live_ocean] downloaded 10136508 bytes from https://liveocean.apl.uw.edu/output/f2024.07.20/ubc.nc to /results/forcing/LiveOcean/downloaded/20240720/low_passed_UBC.nc
2024-07-20 09:03:20,779 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-07-20 09:03:20,995 INFO [collect_NeahBay_ssh] 2024-07-20T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-07-20 09:03:20,998 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-20 09:03:21,008 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 09:03:21,364 INFO [grib_to_netcdf] running in process 700824
2024-07-20 09:03:21,365 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:21,366 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565
2024-07-20 09:03:21,367 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:21,368 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-20 nowcast and forecast runs
2024-07-20 09:03:21,826 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240720/low_passed_UBC.nc
2024-07-20 09:03:21,827 INFO [download_live_ocean] 2024-07-20 Live Ocean file for Salish Sea western boundary downloaded
2024-07-20 09:03:21,829 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-07-20 09:03:21,978 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-07-20'], host='localhost')
2024-07-20 09:03:23,271 INFO [make_turbidity_file] 2024-07-20 Fraser River turbidity file creation complete
2024-07-20 09:03:23,272 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-07-20 09:03:23,610 INFO [make_ssh_files] running in process 700876
2024-07-20 09:03:23,611 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:23,612 INFO [make_ssh_files] publishing log messages to tcp://*:5566
2024-07-20 09:03:23,613 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:23,614 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-07-20 NOAA Neah Bay 06Z observation and forecast values
2024-07-20 09:03:23,956 INFO [make_live_ocean_files] running in process 700891
2024-07-20 09:03:23,958 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:23,958 INFO [make_live_ocean_files] publishing log messages to tcp://*:5567
2024-07-20 09:03:23,959 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:23,962 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-07-20 Live Ocean run
2024-07-20 09:03:27,955 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d16.nc
2024-07-20 09:03:28,117 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d17.nc
2024-07-20 09:03:28,317 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc
2024-07-20 09:03:28,499 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc
2024-07-20 09:03:28,708 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d20.nc
2024-07-20 09:03:28,893 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d21.nc
2024-07-20 09:03:29,075 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d22.nc
2024-07-20 09:03:29,261 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d23.nc
2024-07-20 09:03:29,368 INFO [make_ssh_files] sea surface height boundary file for 2024-07-20 nowcast run created
2024-07-20 09:03:29,371 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-20 09:03:29,377 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-20 09:03:29,382 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-20 09:03:30,078 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m07d20.nc
2024-07-20 09:03:30,705 INFO [make_runoff_file] running in process 700947
2024-07-20 09:03:30,706 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:30,707 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2024-07-20 09:03:30,708 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:30,733 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-20 09:03:30,807 INFO [make_runoff_file] 2024-07-20 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-20 09:03:30,809 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-20 09:03:31,264 INFO [make_v202111_runoff_file] running in process 700945
2024-07-20 09:03:31,266 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:03:31,266 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5566
2024-07-20 09:03:31,267 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:03:31,268 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-19
2024-07-20 09:03:33,748 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-19: /results/forcing/rivers/R202108Dailies_y2024m07d19.nc
2024-07-20 09:03:33,750 INFO [make_v202111_runoff_file] 2024-07-19 runoff file creation completed
2024-07-20 09:03:33,752 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-20 09:03:37,027 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d21.nc
2024-07-20 09:03:40,723 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d22.nc
2024-07-20 09:03:40,737 INFO [grib_to_netcdf] 2024-07-20 NEMO-atmos forcing file for nowcast+ created
2024-07-20 09:03:40,739 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-20 09:04:04,526 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d20.nc
2024-07-20 09:04:04,527 INFO [make_live_ocean_files] 2024-07-20 Live Ocean western boundary conditions files created
2024-07-20 09:04:04,529 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-07-20 09:04:04,543 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-07-20 09:04:04,546 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-07-20 09:04:04,551 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-07-20 09:04:04,556 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-07-20 09:04:04,560 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-07-20 09:04:05,668 INFO [ping_erddap] running in process 701031
2024-07-20 09:04:05,669 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:04:05,670 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-20 09:04:05,672 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:04:05,674 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-07-20 09:04:05,676 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-20 09:04:05,707 INFO [upload_forcing] running in process 701035
2024-07-20 09:04:05,708 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:04:05,709 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-20 09:04:05,710 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:04:05,725 INFO [upload_forcing] running in process 701033
2024-07-20 09:04:05,726 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:04:05,727 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-20 09:04:05,728 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:04:05,730 INFO [upload_forcing] running in process 701036
2024-07-20 09:04:05,731 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:04:05,732 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-20 09:04:05,733 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:04:05,770 INFO [upload_forcing] running in process 701034
2024-07-20 09:04:05,771 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:04:05,772 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-20 09:04:05,774 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:04:07,866 INFO [upload_forcing] nowcast+ 2024-07-20 forcing files upload to optimum-hindcast completed
2024-07-20 09:04:07,870 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-20 09:04:08,041 INFO [upload_forcing] nowcast+ 2024-07-20 forcing files upload to orcinus-nowcast-agrif completed
2024-07-20 09:04:08,045 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-20 09:04:15,260 INFO [upload_forcing] nowcast+ 2024-07-20 forcing files upload to robot.graham completed
2024-07-20 09:04:15,266 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-20 09:04:18,518 INFO [upload_forcing] nowcast+ 2024-07-20 forcing files upload to arbutus.cloud-nowcast completed
2024-07-20 09:04:18,522 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-20 09:04:18,538 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 09:04:19,963 INFO [make_forcing_links] running in process 701070
2024-07-20 09:04:19,964 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:04:19,965 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-20 09:04:19,966 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:04:21,108 INFO [make_forcing_links] nowcast+ 2024-07-20 forcing file links on arbutus.cloud-nowcast created
2024-07-20 09:04:21,113 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-20 09:04:21,128 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast')
2024-07-20 09:04:23,230 INFO [run_NEMO] running in process 19686
2024-07-20 09:04:23,231 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:04:23,233 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-20 09:04:23,301 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:04:24,128 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/20jul24nowcast_2024-07-20T090423.290679-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-20 09:04:24,149 INFO [run_NEMO] nowcast NEMO run for 2024-07-20 on arbutus.cloud-nowcast started
2024-07-20 09:04:24,153 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-20 09:04:24,168 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-20 09:04:26,143 INFO [watch_NEMO] running in process 19939
2024-07-20 09:04:26,146 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:04:26,148 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-20 09:04:26,149 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:04:26,214 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-20 09:09:26,218 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17796676 = 2024-07-20 04:50:00 UTC, 20.1% complete
2024-07-20 09:14:26,220 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17797154 = 2024-07-20 10:08:40 UTC, 42.3% complete
2024-07-20 09:19:26,222 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17797619 = 2024-07-20 15:18:40 UTC, 63.8% complete
2024-07-20 09:24:26,224 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17798120 = 2024-07-20 20:52:40 UTC, 87.0% complete
2024-07-20 09:29:26,230 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-07-20 09:29:26,233 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-20 09:29:26,421 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 09:29:26,429 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 09:29:27,662 INFO [make_forcing_links] running in process 701258
2024-07-20 09:29:27,663 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:29:27,664 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-20 09:29:27,666 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:29:27,667 INFO [download_results] running in process 701259
2024-07-20 09:29:27,668 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:29:27,669 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-20 09:29:27,670 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:29:27,671 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/20jul24 to /results/SalishSea/nowcast-blue.202111
2024-07-20 09:29:28,594 INFO [make_forcing_links] ssh 2024-07-20 forcing file links on arbutus.cloud-nowcast created
2024-07-20 09:29:28,598 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-20 09:29:28,614 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast')
2024-07-20 09:29:31,901 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/20jul24forecast_2024-07-20T092930.960225-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-20 09:29:31,921 INFO [run_NEMO] forecast NEMO run for 2024-07-20 on arbutus.cloud-nowcast started
2024-07-20 09:29:31,926 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-20 09:29:31,938 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-20 09:29:34,750 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-20 09:34:35,368 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17798849 = 2024-07-21 04:58:40 UTC, 13.8% complete
2024-07-20 09:39:35,337 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17799314 = 2024-07-21 10:08:40 UTC, 28.2% complete
2024-07-20 09:44:43,773 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17799793 = 2024-07-21 15:28:00 UTC, 43.0% complete
2024-07-20 09:49:35,395 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17800294 = 2024-07-21 21:02:00 UTC, 58.4% complete
2024-07-20 09:49:53,413 INFO [download_results] nowcast 2024-07-20 results files from arbutus.cloud-nowcast downloaded
2024-07-20 09:49:53,417 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 09:49:53,430 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 09:49:53,435 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-07-19'], host='localhost')
2024-07-20 09:49:53,440 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 09:49:55,409 INFO [make_CHS_currents_file] running in process 701334
2024-07-20 09:49:55,410 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:49:55,411 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-07-20 09:49:55,412 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:49:56,777 INFO [make_plots] running in process 701333
2024-07-20 09:49:56,778 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:49:56,779 INFO [make_plots] publishing log messages to tcp://*:5566
2024-07-20 09:49:56,780 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:49:56,798 INFO [make_plots] running in process 701331
2024-07-20 09:49:56,821 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 09:49:56,827 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-20 09:49:56,828 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-20 09:49:57,442 INFO [make_plots] preparing render list for 2024-07-20 NEMO nowcast-blue research figures
2024-07-20 09:49:57,445 INFO [make_plots] preparing render list for 2024-07-19 NEMO nowcast-blue comparison figures
2024-07-20 09:49:58,608 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-blue research render list
2024-07-20 09:49:59,283 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-blue research render list
2024-07-20 09:49:59,284 INFO [make_plots] added Currents_sections_and_surface figure to 2024-07-20 NEMO nowcast-blue research render list
2024-07-20 09:49:59,285 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-07-20 NEMO nowcast-blue research render list
2024-07-20 09:49:59,287 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-07-20 NEMO nowcast-blue research render list
2024-07-20 09:49:59,288 INFO [make_plots] starting to render nemo nowcast research 20jul24 figures
2024-07-20 09:50:16,395 INFO [make_plots] added SH_wind figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,395 INFO [make_plots] added Compare_VENUS_East figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,396 INFO [make_plots] added Compare_VENUS_Central figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,397 INFO [make_plots] added Central_ADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,399 INFO [make_plots] added Central_depavADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,399 INFO [make_plots] added Central_timeavADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,400 INFO [make_plots] added East_ADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,401 INFO [make_plots] added East_depavADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,402 INFO [make_plots] added East_timeavADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,403 INFO [make_plots] added ddl_ADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,404 INFO [make_plots] added ddl_depavADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,404 INFO [make_plots] added ddl_timeavADCP figure to 2024-07-19 NEMO nowcast-blue comparison render list
2024-07-20 09:50:16,405 INFO [make_plots] starting to render nemo nowcast comparison 19jul24 figures
2024-07-20 09:50:25,652 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-20 for nowcast
2024-07-20 09:50:25,656 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-20 09:51:08,180 INFO [make_plots] finished rendering nemo nowcast research 20jul24 figures
2024-07-20 09:51:08,183 INFO [make_plots] nemo research plots for 2024-07-20 nowcast completed
2024-07-20 09:51:08,196 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-20 09:51:39,025 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available
2024-07-20 09:51:39,600 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-20 09:51:40,094 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-20 09:51:40,584 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-20 09:51:41,077 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-20 09:51:41,566 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-20 09:51:42,054 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-20 09:51:42,546 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-20 09:51:43,035 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-20 09:51:43,523 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-20 09:51:43,525 INFO [make_plots] finished rendering nemo nowcast comparison 19jul24 figures
2024-07-20 09:51:43,614 INFO [make_plots] nemo comparison plots for 2024-07-19 nowcast completed
2024-07-20 09:51:43,617 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-20 09:54:34,704 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17800761 = 2024-07-22 02:13:20 UTC, 72.9% complete
2024-07-20 09:59:34,702 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17801231 = 2024-07-22 07:26:40 UTC, 87.4% complete
2024-07-20 10:00:12,077 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 10:00:12,275 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-03-26'], host='localhost')
2024-07-20 10:00:12,282 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 10:00:12,287 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 10:00:13,500 INFO [download_results] running in process 701462
2024-07-20 10:00:13,501 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:00:13,502 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-20 10:00:13,503 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:00:13,504 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26mar18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 10:00:17,811 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 10:00:58,753 INFO [download_results] hindcast 2018-03-26 results files from optimum-hindcast downloaded
2024-07-20 10:00:58,764 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 10:00:58,781 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-03-26'], host='localhost')
2024-07-20 10:01:00,012 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 10:01:00,032 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-mar', 'robot.graham'], host='localhost')
2024-07-20 10:01:01,155 INFO [archive_tarball] running in process 701509
2024-07-20 10:01:01,156 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:01:01,157 INFO [archive_tarball] publishing log messages to tcp://*:5565
2024-07-20 10:01:01,159 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:01:01,159 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar18.tar from /ocean/sallen/timbucktwo/oxygen/*mar18/
2024-07-20 10:04:34,700 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17801640 = 2024-07-22 11:59:20 UTC, 100.0% complete
2024-07-20 10:06:18,775 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar18.index from /ocean/dlatorne/oxygen-mar18.tar
2024-07-20 10:06:18,856 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-mar18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-20 10:09:34,703 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-07-20 10:09:34,706 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-20 10:09:34,886 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast')
2024-07-20 10:09:34,893 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast')
2024-07-20 10:09:34,898 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:09:34,904 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:09:34,909 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:09:34,913 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:09:34,919 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:09:36,488 INFO [upload_forcing] running in process 701573
2024-07-20 10:09:36,489 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:36,490 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-20 10:09:36,492 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:36,500 INFO [upload_forcing] running in process 701571
2024-07-20 10:09:36,502 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:36,503 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-20 10:09:36,504 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:36,505 INFO [upload_forcing] running in process 701572
2024-07-20 10:09:36,506 INFO [upload_forcing] running in process 701570
2024-07-20 10:09:36,507 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:36,508 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:36,509 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-20 10:09:36,510 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-20 10:09:36,511 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:36,512 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:36,513 INFO [download_results] running in process 701574
2024-07-20 10:09:36,514 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:36,515 INFO [download_results] publishing log messages to tcp://*:5570
2024-07-20 10:09:36,516 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:36,517 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/20jul24 to /results/SalishSea/forecast.202111
2024-07-20 10:09:37,160 INFO [upload_forcing] turbidity 2024-07-20 forcing files upload to optimum-hindcast completed
2024-07-20 10:09:37,165 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-20 10:09:37,246 INFO [upload_forcing] turbidity 2024-07-20 forcing files upload to orcinus-nowcast-agrif completed
2024-07-20 10:09:37,251 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-20 10:09:37,272 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:09:37,532 INFO [upload_forcing] turbidity 2024-07-20 forcing files upload to arbutus.cloud-nowcast completed
2024-07-20 10:09:37,536 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-20 10:09:37,559 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:09:37,851 INFO [make_ww3_wind_file] running in process 20893
2024-07-20 10:09:37,852 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:37,854 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-20 10:09:37,910 INFO [make_ww3_current_file] running in process 20904
2024-07-20 10:09:37,911 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:37,912 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-20 10:09:37,982 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:37,983 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-20 forecast run
2024-07-20 10:09:37,986 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:37,987 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-20 forecast run
2024-07-20 10:09:38,608 INFO [make_forcing_links] running in process 701605
2024-07-20 10:09:38,609 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:38,610 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2024-07-20 10:09:38,611 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:38,936 INFO [make_forcing_links] running in process 701607
2024-07-20 10:09:38,938 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:38,939 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2024-07-20 10:09:38,940 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:39,497 INFO [make_forcing_links] nowcast-agrif 2024-07-20 forcing file links on orcinus-nowcast-agrif created
2024-07-20 10:09:39,501 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-20 10:09:39,522 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:09:40,616 INFO [upload_forcing] turbidity 2024-07-20 forcing files upload to robot.graham completed
2024-07-20 10:09:40,621 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-20 10:09:40,897 INFO [run_NEMO_agrif] running in process 701621
2024-07-20 10:09:40,898 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:40,899 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5566
2024-07-20 10:09:40,901 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:42,693 INFO [make_forcing_links] nowcast-green 2024-07-20 forcing file links on arbutus.cloud-nowcast created
2024-07-20 10:09:42,697 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-20 10:09:42,714 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast')
2024-07-20 10:09:48,234 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/20jul24nowcast-green_2024-07-20T100947.164587-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-20 10:09:48,255 INFO [run_NEMO] nowcast-green NEMO run for 2024-07-20 on arbutus.cloud-nowcast started
2024-07-20 10:09:48,260 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-20 10:09:48,280 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-07-20 10:09:48,456 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-20 forecast run
2024-07-20 10:09:48,460 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-20 10:09:52,063 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-20 10:09:55,139 INFO [run_NEMO_agrif] job id for 20jul24nowcast-agrif: 11101962.orca2.ibb
2024-07-20 10:09:55,140 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-07-20 queued on orcinus-nowcast-agrif
2024-07-20 10:09:55,145 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-07-20 10:09:55,162 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11101962.orca2.ibb'], host='localhost')
2024-07-20 10:09:56,466 INFO [watch_NEMO_agrif] running in process 701640
2024-07-20 10:09:56,467 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:09:56,468 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5566
2024-07-20 10:09:56,470 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:09:57,256 INFO [watch_NEMO_agrif] watching 20jul24nowcast-agrif job 11101962 on orcinus-nowcast-agrif
2024-07-20 10:09:57,490 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif job 11101962 is queued on orcinus-nowcast-agrif
2024-07-20 10:10:35,784 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-20 forecast run
2024-07-20 10:10:35,787 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-20 10:10:35,812 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast')
2024-07-20 10:10:46,897 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-20 10:10:46,919 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-20 10:10:57,695 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif job 11101962 is queued on orcinus-nowcast-agrif
2024-07-20 10:10:58,744 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-20 10:11:57,901 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif job 11101962 is queued on orcinus-nowcast-agrif
2024-07-20 10:12:58,955 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-07-20 10:13:09,907 INFO [archive_tarball] hindcast *mar18 results files archived to robot.graham
2024-07-20 10:13:09,912 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-20 10:14:52,596 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17796406 = 2024-07-20 01:50:00 UTC, 7.6% complete
2024-07-20 10:15:58,787 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete
2024-07-20 10:17:59,390 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7817231 = 2024-07-20 02:06:40 UTC, 8.8% complete
2024-07-20 10:19:52,593 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17796598 = 2024-07-20 03:58:00 UTC, 16.5% complete
2024-07-20 10:20:58,795 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 183, 42.4% complete
2024-07-20 10:22:59,865 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7817444 = 2024-07-20 04:28:40 UTC, 18.7% complete
2024-07-20 10:24:52,579 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17796786 = 2024-07-20 06:03:20 UTC, 25.2% complete
2024-07-20 10:25:58,801 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 273, 63.2% complete
2024-07-20 10:28:00,339 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7817657 = 2024-07-20 06:50:40 UTC, 28.5% complete
2024-07-20 10:29:52,579 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17796971 = 2024-07-20 08:06:40 UTC, 33.8% complete
2024-07-20 10:30:58,781 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 363, 84.0% complete
2024-07-20 10:33:00,816 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7817868 = 2024-07-20 09:11:20 UTC, 38.3% complete
2024-07-20 10:34:52,230 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17797155 = 2024-07-20 10:09:20 UTC, 42.3% complete
2024-07-20 10:35:58,430 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-20 10:38:01,252 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7818082 = 2024-07-20 11:34:00 UTC, 48.2% complete
2024-07-20 10:39:15,401 INFO [download_results] forecast 2024-07-20 results files from arbutus.cloud-nowcast downloaded
2024-07-20 10:39:15,406 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 10:39:15,427 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:39:17,796 INFO [make_CHS_currents_file] running in process 701910
2024-07-20 10:39:17,798 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:39:17,799 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-07-20 10:39:17,800 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:39:51,828 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17797346 = 2024-07-20 12:16:40 UTC, 51.2% complete
2024-07-20 10:39:58,289 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-20 for forecast
2024-07-20 10:39:58,293 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-20 10:39:58,313 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:39:59,434 INFO [update_forecast_datasets] running in process 701935
2024-07-20 10:39:59,435 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:39:59,436 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-07-20 10:39:59,437 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:39:59,439 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/20jul24 run
2024-07-20 10:39:59,539 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/20jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-20 10:39:59,540 INFO [update_forecast_datasets] nemo 2024-07-20 forecast rolling forecast datasets updated
2024-07-20 10:39:59,542 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-20 10:39:59,562 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-20 10:39:59,567 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:39:59,572 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:40:00,766 INFO [ping_erddap] running in process 701940
2024-07-20 10:40:00,767 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:40:00,767 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-20 10:40:00,768 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:40:00,778 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-20 10:40:00,779 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-20 10:40:01,928 INFO [make_surface_current_tiles] running in process 701943
2024-07-20 10:40:01,929 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:40:01,930 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-07-20 10:40:01,931 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:40:01,933 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-20 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/20jul24
2024-07-20 10:40:03,725 INFO [make_plots] running in process 701942
2024-07-20 10:40:03,727 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:40:03,728 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-20 10:40:03,729 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:40:04,482 INFO [make_plots] preparing render list for 2024-07-20 NEMO forecast publish figures
2024-07-20 10:40:04,606 INFO [make_plots] added Website_thumbnail figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,608 INFO [make_plots] added Threshold_website figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,609 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,610 INFO [make_plots] added Boundary Bay figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,611 INFO [make_plots] added Campbell River figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,611 INFO [make_plots] added Cherry Point figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,612 INFO [make_plots] added Friday Harbor figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,613 INFO [make_plots] added Halfmoon Bay figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,614 INFO [make_plots] added Nanaimo figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,615 INFO [make_plots] added Neah Bay figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,615 INFO [make_plots] added New Westminster figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,616 INFO [make_plots] added Patricia Bay figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,617 INFO [make_plots] added Point Atkinson figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,617 INFO [make_plots] added Port Renfrew figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,618 INFO [make_plots] added Sand Heads figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,619 INFO [make_plots] added Sandy Cove figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,620 INFO [make_plots] added Squamish figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,621 INFO [make_plots] added Victoria figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,623 INFO [make_plots] added Woodwards Landing figure to 2024-07-20 NEMO forecast publish render list
2024-07-20 10:40:04,623 INFO [make_plots] starting to render nemo forecast publish 20jul24 figures
2024-07-20 10:40:58,141 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-20 10:40:58,143 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-20 10:40:58,330 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 10:40:58,339 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast')
2024-07-20 10:40:59,958 INFO [download_wwatch3_results] running in process 702247
2024-07-20 10:40:59,958 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:40:59,959 INFO [download_wwatch3_results] publishing log messages to tcp://*:5568
2024-07-20 10:40:59,961 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:41:00,372 INFO [run_ww3] running in process 22329
2024-07-20 10:41:00,373 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:41:00,374 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-07-20 10:41:00,375 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:41:00,378 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-07-20T104100.355169-0700
2024-07-20 10:41:00,379 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-07-20T104100.355169-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-07-20 10:41:00,398 INFO [run_ww3] forecast WaveWatch3 run for 2024-07-20 on arbutus.cloud-nowcast started
2024-07-20 10:41:00,401 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-20 10:41:00,436 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-20 10:41:02,443 INFO [watch_ww3] running in process 22483
2024-07-20 10:41:02,444 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 10:41:02,445 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-07-20 10:41:02,446 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-20 10:41:02,480 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-20 10:42:57,086 INFO [download_wwatch3_results] nowcast 2024-07-20 results files from arbutus.cloud-nowcast downloaded
2024-07-20 10:42:57,090 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-20 10:43:01,720 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7818295 = 2024-07-20 13:56:00 UTC, 58.1% complete
2024-07-20 10:44:51,906 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17797533 = 2024-07-20 14:21:20 UTC, 59.8% complete
2024-07-20 10:46:02,488 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 81, 12.5% complete
2024-07-20 10:48:02,164 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7818506 = 2024-07-20 16:16:40 UTC, 67.9% complete
2024-07-20 10:49:51,974 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17797718 = 2024-07-20 16:24:40 UTC, 68.4% complete
2024-07-20 10:50:31,685 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/20jul24
2024-07-20 10:51:02,522 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 174, 26.9% complete
2024-07-20 10:52:57,642 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-20 10:52:57,643 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-20 10:53:02,642 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7818720 = 2024-07-20 18:39:20 UTC, 77.8% complete
2024-07-20 10:53:33,128 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:54:51,967 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17797901 = 2024-07-20 18:26:40 UTC, 76.9% complete
2024-07-20 10:55:02,190 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/20jul24
2024-07-20 10:55:02,202 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-20 forecast into /results/nowcast-sys/figures/surface_currents/forecast/20jul24
2024-07-20 10:55:02,203 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-20 forecast completed
2024-07-20 10:55:02,516 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-20 10:55:19,518 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:55:19,872 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:55:54,787 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:56:02,506 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 264, 40.7% complete
2024-07-20 10:57:04,123 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:57:39,701 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:58:03,118 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7818931 = 2024-07-20 21:00:00 UTC, 87.5% complete
2024-07-20 10:58:15,509 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:58:51,120 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:59:25,525 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 10:59:51,965 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17798081 = 2024-07-20 20:26:40 UTC, 85.2% complete
2024-07-20 11:00:01,339 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 11:00:36,724 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 11:00:37,072 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 11:01:02,502 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 354, 54.6% complete
2024-07-20 11:01:12,427 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 11:01:46,755 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-15 00:00:00Z to 2024-07-22 11:55:00Z
2024-07-20 11:02:22,193 INFO [make_plots] finished rendering nemo forecast publish 20jul24 figures
2024-07-20 11:02:22,196 INFO [make_plots] nemo publish plots for 2024-07-20 forecast completed
2024-07-20 11:02:22,205 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-20 11:02:22,377 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:02:25,276 INFO [make_feeds] running in process 703858
2024-07-20 11:02:25,277 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:02:25,278 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-07-20 11:02:25,280 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:02:53,310 INFO [make_feeds] ATOM feeds for 2024-07-20 forecast run completed
2024-07-20 11:02:53,313 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-20 11:03:03,555 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7819145 = 2024-07-20 23:22:40 UTC, 97.5% complete
2024-07-20 11:04:51,991 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17798262 = 2024-07-20 22:27:20 UTC, 93.6% complete
2024-07-20 11:06:02,502 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 450, 69.4% complete
2024-07-20 11:08:03,983 INFO [watch_NEMO_agrif] 20jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7819200 = 2024-07-20 23:59:20 UTC, 100.0% complete
2024-07-20 11:09:51,959 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17798400 = 2024-07-20 23:59:20 UTC, 100.0% complete
2024-07-20 11:11:02,491 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 540, 83.3% complete
2024-07-20 11:13:04,204 INFO [watch_NEMO_agrif] NEMO AGRIF run 11101962.orca2.ibb on orcinus-nowcast-agrif completed
2024-07-20 11:13:04,209 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-07-20 11:13:04,534 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:13:06,035 INFO [download_results] running in process 703990
2024-07-20 11:13:06,036 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:13:06,038 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-20 11:13:06,039 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:13:06,040 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/20jul24 to /results/SalishSea/nowcast-agrif.201702
2024-07-20 11:13:29,255 INFO [download_results] nowcast-agrif 2024-07-20 results files from orcinus-nowcast-agrif downloaded
2024-07-20 11:13:29,263 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 11:13:29,441 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:13:33,389 INFO [make_plots] running in process 703998
2024-07-20 11:13:33,390 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:13:33,392 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-20 11:13:33,394 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:13:33,961 INFO [make_plots] preparing render list for 2024-07-20 NEMO nowcast-agrif research figures
2024-07-20 11:13:33,963 INFO [make_plots] added baynes_sound_surface figure to 2024-07-20 NEMO nowcast-agrif research render list
2024-07-20 11:13:33,964 INFO [make_plots] starting to render nemo nowcast-agrif research 20jul24 figures
2024-07-20 11:13:45,154 INFO [make_plots] finished rendering nemo nowcast-agrif research 20jul24 figures
2024-07-20 11:13:45,157 INFO [make_plots] nemo research plots for 2024-07-20 nowcast-agrif completed
2024-07-20 11:13:45,161 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-20 11:14:51,970 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-07-20 11:14:51,975 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-20 11:14:52,294 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:14:53,735 INFO [download_results] running in process 704036
2024-07-20 11:14:53,737 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:14:53,738 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-20 11:14:53,739 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:14:53,740 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/20jul24 to /results2/SalishSea/nowcast-green.202111
2024-07-20 11:16:02,998 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 633, 97.7% complete
2024-07-20 11:21:03,147 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-20 11:21:03,189 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-20 11:21:03,553 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:21:04,854 INFO [download_wwatch3_results] running in process 704100
2024-07-20 11:21:04,855 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:21:04,857 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-07-20 11:21:04,858 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:25:22,315 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 11:25:22,636 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-04-01'], host='localhost')
2024-07-20 11:25:22,643 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 11:25:22,649 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 11:25:23,887 INFO [download_results] running in process 704120
2024-07-20 11:25:23,889 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:25:23,890 INFO [download_results] publishing log messages to tcp://*:5568
2024-07-20 11:25:23,891 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:25:23,892 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01apr18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 11:25:28,296 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 11:26:04,631 INFO [download_results] hindcast 2018-04-01 results files from optimum-hindcast downloaded
2024-07-20 11:26:04,642 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 11:26:04,815 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-04-01'], host='localhost')
2024-07-20 11:26:06,059 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 11:27:34,719 INFO [download_wwatch3_results] forecast 2024-07-20 results files from arbutus.cloud-nowcast downloaded
2024-07-20 11:27:34,723 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-20 11:27:34,895 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:27:36,249 INFO [update_forecast_datasets] running in process 704169
2024-07-20 11:27:36,250 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:27:36,251 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-07-20 11:27:36,252 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:27:36,254 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/20jul24 run
2024-07-20 11:27:36,255 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/20jul24 run
2024-07-20 11:27:36,277 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/20jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-20 11:27:36,278 INFO [update_forecast_datasets] wwatch3 2024-07-20 forecast rolling forecast datasets updated
2024-07-20 11:27:36,280 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-20 11:27:36,451 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-20 11:27:37,698 INFO [ping_erddap] running in process 704174
2024-07-20 11:27:37,699 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:27:37,700 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-20 11:27:37,702 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:27:37,703 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-20 11:27:37,706 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-20 11:27:37,878 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:27:41,425 INFO [make_plots] running in process 704179
2024-07-20 11:27:41,427 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:27:41,428 INFO [make_plots] publishing log messages to tcp://*:5562
2024-07-20 11:27:41,429 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:27:41,430 INFO [make_plots] preparing render list for 2024-07-20 WaveWatch3 forecast publish figures
2024-07-20 11:27:41,431 INFO [make_plots] added Halibut Bank figure to 2024-07-20 WaveWatch3 forecast publish render list
2024-07-20 11:27:41,433 INFO [make_plots] added Sentry Shoal figure to 2024-07-20 WaveWatch3 forecast publish render list
2024-07-20 11:27:41,434 INFO [make_plots] starting to render wwatch3 forecast publish 20jul24 figures
2024-07-20 11:27:41,726 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-20 11:27:48,887 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-20 11:27:55,032 INFO [make_plots] finished rendering wwatch3 forecast publish 20jul24 figures
2024-07-20 11:27:55,033 INFO [make_plots] wwatch3 publish plots for 2024-07-20 forecast completed
2024-07-20 11:27:55,036 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-20 11:52:00,842 INFO [download_results] nowcast-green 2024-07-20 results files from arbutus.cloud-nowcast downloaded
2024-07-20 11:52:00,849 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 11:52:01,026 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:52:01,032 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-07-20 11:52:01,036 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:52:01,039 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:52:01,043 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-07-20'], host='localhost')
2024-07-20 11:52:02,585 INFO [ping_erddap] running in process 704271
2024-07-20 11:52:02,587 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:52:02,588 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-20 11:52:02,588 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:52:02,593 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-07-20 11:52:02,595 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-20 11:52:03,566 INFO [make_averaged_dataset] running in process 704272
2024-07-20 11:52:03,568 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:52:03,569 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2024-07-20 11:52:03,569 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:52:03,570 INFO [make_averaged_dataset] creating day-averaged dataset for 20-Jul-2024 biology
2024-07-20 11:52:03,576 INFO [make_averaged_dataset] running in process 704274
2024-07-20 11:52:03,577 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:52:03,579 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2024-07-20 11:52:03,579 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:52:03,580 INFO [make_averaged_dataset] creating day-averaged dataset for 20-Jul-2024 physics
2024-07-20 11:52:03,593 INFO [make_averaged_dataset] running in process 704273
2024-07-20 11:52:03,594 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:52:03,595 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2024-07-20 11:52:03,596 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:52:03,597 INFO [make_averaged_dataset] creating day-averaged dataset for 20-Jul-2024 chemistry
2024-07-20 11:52:04,666 INFO [make_plots] running in process 704269
2024-07-20 11:52:04,668 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 11:52:04,669 INFO [make_plots] publishing log messages to tcp://*:5568
2024-07-20 11:52:04,670 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-20 11:52:05,184 INFO [make_plots] preparing render list for 2024-07-20 NEMO nowcast-green research figures
2024-07-20 11:52:06,910 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:07,632 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:08,349 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:09,091 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:09,839 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:10,595 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:11,335 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:12,212 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:13,073 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:13,934 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:14,786 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:15,558 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:16,256 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:16,256 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:16,257 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:16,258 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-07-20 NEMO nowcast-green research render list
2024-07-20 11:52:16,258 INFO [make_plots] starting to render nemo nowcast-green research 20jul24 figures
2024-07-20 11:52:44,106 INFO [make_averaged_dataset] day-averaged dataset for 20-Jul-2024 physics created
2024-07-20 11:52:44,108 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-20 11:53:17,474 INFO [make_averaged_dataset] day-averaged dataset for 20-Jul-2024 chemistry created
2024-07-20 11:53:17,478 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-20 11:53:32,495 INFO [make_averaged_dataset] day-averaged dataset for 20-Jul-2024 biology created
2024-07-20 11:53:32,498 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-20 12:50:32,625 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 12:50:33,008 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-04-06'], host='localhost')
2024-07-20 12:50:33,017 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 12:50:33,022 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 12:50:35,363 INFO [download_results] running in process 704779
2024-07-20 12:50:35,364 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 12:50:35,365 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-20 12:50:35,366 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 12:50:35,368 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06apr18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 12:50:39,603 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 12:51:25,490 INFO [download_results] hindcast 2018-04-06 results files from optimum-hindcast downloaded
2024-07-20 12:51:25,514 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 12:51:25,720 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-04-06'], host='localhost')
2024-07-20 12:51:27,088 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 13:00:45,727 INFO [make_plots] finished rendering nemo nowcast-green research 20jul24 figures
2024-07-20 13:00:45,739 INFO [make_plots] nemo research plots for 2024-07-20 nowcast-green completed
2024-07-20 13:00:45,799 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-20 14:20:44,302 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 14:20:44,648 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-04-11'], host='localhost')
2024-07-20 14:20:44,656 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 14:20:44,661 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 14:20:45,888 INFO [download_results] running in process 705281
2024-07-20 14:20:45,889 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 14:20:45,890 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-20 14:20:45,892 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 14:20:45,893 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11apr18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 14:20:50,088 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 14:21:25,775 INFO [download_results] hindcast 2018-04-11 results files from optimum-hindcast downloaded
2024-07-20 14:21:25,786 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 14:21:25,993 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-04-11'], host='localhost')
2024-07-20 14:21:27,195 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 15:02:45,273 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240720/18/
2024-07-20 15:02:45,274 INFO [collect_weather] 2024-07-20 2.5km weather forecast 18 collection complete
2024-07-20 15:02:45,277 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-20 15:02:45,633 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-07-20 15:02:45,640 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-07-20 15:02:45,643 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-07-20 15:02:45,649 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-07-21'], host='localhost')
2024-07-20 15:02:46,160 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240720/18/
2024-07-20 15:02:46,161 INFO [crop_gribs] 2024-07-20 18 GRIBs cropping complete
2024-07-20 15:02:46,163 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-20 15:02:46,934 INFO [download_weather] running in process 705691
2024-07-20 15:02:46,935 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 15:02:46,936 INFO [download_weather] publishing log messages to tcp://*:5564
2024-07-20 15:02:46,938 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-20 15:02:46,939 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240720
2024-07-20 15:02:46,940 INFO [download_weather] running in process 705693
2024-07-20 15:02:46,941 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 15:02:46,942 INFO [download_weather] publishing log messages to tcp://*:5565
2024-07-20 15:02:46,942 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-20 15:02:46,943 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240720
2024-07-20 15:02:46,943 INFO [collect_weather] running in process 705694
2024-07-20 15:02:46,944 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 15:02:46,945 INFO [collect_weather] publishing log messages to tcp://*:5566
2024-07-20 15:02:46,945 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-20 15:02:47,087 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-07-20 15:02:48,256 INFO [crop_gribs] running in process 705695
2024-07-20 15:02:48,260 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 15:02:48,261 INFO [crop_gribs] publishing log messages to tcp://*:5567
2024-07-20 15:02:48,262 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-20 15:02:48,263 INFO [crop_gribs] cropping 2024-07-21 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-07-20 15:02:48,277 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240721/00/
2024-07-20 15:07:27,559 INFO [download_weather] 2024-07-20 1km weather forecast 00 downloads complete
2024-07-20 15:07:27,562 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-20 15:07:38,223 INFO [download_weather] 2024-07-20 1km weather forecast 12 downloads complete
2024-07-20 15:07:38,227 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-20 15:45:54,556 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 15:45:54,911 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-04-16'], host='localhost')
2024-07-20 15:45:54,919 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 15:45:54,925 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 15:45:56,121 INFO [download_results] running in process 705828
2024-07-20 15:45:56,123 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 15:45:56,124 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-20 15:45:56,125 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 15:45:56,127 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16apr18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 15:46:00,389 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 15:46:35,980 INFO [download_results] hindcast 2018-04-16 results files from optimum-hindcast downloaded
2024-07-20 15:46:35,990 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 15:46:36,192 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-04-16'], host='localhost')
2024-07-20 15:46:37,400 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 17:11:04,603 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 17:11:04,958 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-04-21'], host='localhost')
2024-07-20 17:11:04,966 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 17:11:04,971 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 17:11:06,240 INFO [download_results] running in process 706258
2024-07-20 17:11:06,242 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 17:11:06,243 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-20 17:11:06,244 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 17:11:06,245 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21apr18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 17:11:10,644 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 17:11:46,057 INFO [download_results] hindcast 2018-04-21 results files from optimum-hindcast downloaded
2024-07-20 17:11:46,068 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 17:11:46,270 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-04-21'], host='localhost')
2024-07-20 17:11:47,533 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 18:41:15,378 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 18:41:15,739 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-04-26'], host='localhost')
2024-07-20 18:41:15,747 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 18:41:15,753 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 18:41:17,019 INFO [download_results] running in process 707585
2024-07-20 18:41:17,021 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 18:41:17,022 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-20 18:41:17,023 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 18:41:17,024 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26apr18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 18:41:21,304 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 18:41:56,933 INFO [download_results] hindcast 2018-04-26 results files from optimum-hindcast downloaded
2024-07-20 18:41:56,943 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 18:41:57,145 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-04-26'], host='localhost')
2024-07-20 18:41:58,221 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 18:41:58,421 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-apr', 'robot.graham'], host='localhost')
2024-07-20 18:41:59,534 INFO [archive_tarball] running in process 707636
2024-07-20 18:41:59,536 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 18:41:59,537 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-07-20 18:41:59,538 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-20 18:41:59,539 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-apr18.tar from /ocean/sallen/timbucktwo/oxygen/*apr18/
2024-07-20 18:47:56,492 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-apr18.index from /ocean/dlatorne/oxygen-apr18.tar
2024-07-20 18:47:56,749 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-apr18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-20 18:57:09,484 INFO [archive_tarball] hindcast *apr18 results files archived to robot.graham
2024-07-20 18:57:09,488 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-20 20:06:25,695 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 20:06:26,049 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-05-01'], host='localhost')
2024-07-20 20:06:26,056 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 20:06:26,062 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 20:06:27,535 INFO [download_results] running in process 707929
2024-07-20 20:06:27,537 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 20:06:27,538 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-20 20:06:27,539 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 20:06:27,540 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01may18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 20:06:31,938 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 20:07:07,426 INFO [download_results] hindcast 2018-05-01 results files from optimum-hindcast downloaded
2024-07-20 20:07:07,436 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 20:07:07,636 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-05-01'], host='localhost')
2024-07-20 20:07:08,867 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 21:04:29,840 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240721/00/
2024-07-20 21:04:29,842 INFO [collect_weather] 2024-07-21 2.5km weather forecast 00 collection complete
2024-07-20 21:04:29,845 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-20 21:04:30,201 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-07-20 21:04:30,207 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-07-21'], host='localhost')
2024-07-20 21:04:31,364 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240721/00/
2024-07-20 21:04:31,366 INFO [crop_gribs] 2024-07-21 00 GRIBs cropping complete
2024-07-20 21:04:31,368 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-20 21:04:31,450 INFO [collect_weather] running in process 708170
2024-07-20 21:04:31,452 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 21:04:31,453 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-20 21:04:31,454 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-20 21:04:31,470 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-07-20 21:04:32,245 INFO [crop_gribs] running in process 708172
2024-07-20 21:04:32,247 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 21:04:32,249 INFO [crop_gribs] publishing log messages to tcp://*:5565
2024-07-20 21:04:32,250 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-20 21:04:32,251 INFO [crop_gribs] cropping 2024-07-21 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-07-20 21:04:32,265 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240721/06/
2024-07-20 21:31:36,195 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 21:31:36,559 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-05-06'], host='localhost')
2024-07-20 21:31:36,567 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 21:31:36,572 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 21:31:37,780 INFO [download_results] running in process 708460
2024-07-20 21:31:37,781 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 21:31:37,782 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-20 21:31:37,784 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 21:31:37,785 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06may18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 21:31:41,921 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 21:32:17,638 INFO [download_results] hindcast 2018-05-06 results files from optimum-hindcast downloaded
2024-07-20 21:32:17,647 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 21:32:17,808 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-05-06'], host='localhost')
2024-07-20 21:32:19,133 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-20 22:56:46,390 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-20 22:56:46,746 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-05-11'], host='localhost')
2024-07-20 22:56:46,753 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 22:56:46,760 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-20 22:56:47,980 INFO [download_results] running in process 708953
2024-07-20 22:56:47,981 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-20 22:56:47,983 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-20 22:56:47,984 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-20 22:56:47,985 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11may18 to /ocean/sallen/timbucktwo/oxygen
2024-07-20 22:56:52,268 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-20 22:57:27,930 INFO [download_results] hindcast 2018-05-11 results files from optimum-hindcast downloaded
2024-07-20 22:57:27,939 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-20 22:57:28,137 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-05-11'], host='localhost')
2024-07-20 22:57:29,258 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 00:26:56,685 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 00:26:57,057 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-05-16'], host='localhost')
2024-07-21 00:26:57,064 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 00:26:57,070 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 00:26:58,326 INFO [download_results] running in process 710133
2024-07-21 00:26:58,327 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 00:26:58,328 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 00:26:58,329 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 00:26:58,331 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16may18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 00:27:02,556 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 00:27:38,653 INFO [download_results] hindcast 2018-05-16 results files from optimum-hindcast downloaded
2024-07-21 00:27:38,664 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 00:27:38,866 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-05-16'], host='localhost')
2024-07-21 00:27:40,198 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 01:52:06,841 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 01:52:07,196 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-05-21'], host='localhost')
2024-07-21 01:52:07,203 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 01:52:07,209 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 01:52:08,406 INFO [download_results] running in process 710734
2024-07-21 01:52:08,407 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 01:52:08,409 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 01:52:08,410 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 01:52:08,411 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21may18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 01:52:12,680 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 01:52:48,164 INFO [download_results] hindcast 2018-05-21 results files from optimum-hindcast downloaded
2024-07-21 01:52:48,176 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 01:52:48,379 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-05-21'], host='localhost')
2024-07-21 01:52:49,536 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 03:02:06,188 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240721/06/
2024-07-21 03:02:06,190 INFO [collect_weather] 2024-07-21 2.5km weather forecast 06 collection complete
2024-07-21 03:02:06,192 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-21 03:02:06,541 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,547 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,553 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,557 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,561 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,566 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,572 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,579 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,586 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,590 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,602 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,606 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,614 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,619 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,636 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-07-21 03:02:06,651 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-07-21 03:02:06,663 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-07-21 03:02:06,675 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-07-20'], host='localhost')
2024-07-21 03:02:06,687 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-07-21 03:02:06,699 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-07-21 03:02:06,718 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-07-21 03:02:07,241 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240721/06/
2024-07-21 03:02:07,241 INFO [crop_gribs] 2024-07-21 06 GRIBs cropping complete
2024-07-21 03:02:07,243 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-21 03:02:07,845 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-07-21 03:02:09,610 INFO [collect_weather] running in process 711076
2024-07-21 03:02:09,611 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:09,612 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-21 03:02:09,613 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:09,634 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-07-21 03:02:09,635 INFO [collect_NeahBay_ssh] running in process 711075
2024-07-21 03:02:09,635 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:09,637 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-07-21 03:02:09,638 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:09,639 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-07-21
2024-07-21 03:02:10,618 INFO [collect_river_data] running in process 711059
2024-07-21 03:02:10,619 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,620 INFO [collect_river_data] publishing log messages to tcp://*:5565
2024-07-21 03:02:10,621 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,622 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-07-20
2024-07-21 03:02:10,720 INFO [collect_river_data] running in process 711063
2024-07-21 03:02:10,721 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,722 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-21 03:02:10,723 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,723 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-07-20
2024-07-21 03:02:10,735 INFO [collect_river_data] running in process 711061
2024-07-21 03:02:10,736 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,737 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-21 03:02:10,738 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,739 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-07-20
2024-07-21 03:02:10,772 INFO [collect_river_data] running in process 711058
2024-07-21 03:02:10,773 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,775 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-21 03:02:10,776 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,776 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-07-20
2024-07-21 03:02:10,781 INFO [collect_river_data] running in process 711060
2024-07-21 03:02:10,783 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,785 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-21 03:02:10,786 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,800 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-07-20
2024-07-21 03:02:10,801 INFO [collect_river_data] running in process 711070
2024-07-21 03:02:10,802 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,803 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-21 03:02:10,805 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,806 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-07-20
2024-07-21 03:02:10,807 INFO [collect_river_data] running in process 711055
2024-07-21 03:02:10,807 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,808 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-07-21 03:02:10,809 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,810 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-07-20
2024-07-21 03:02:10,813 INFO [collect_river_data] running in process 711064
2024-07-21 03:02:10,814 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,815 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-07-21 03:02:10,817 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,818 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-07-20
2024-07-21 03:02:10,823 INFO [collect_river_data] running in process 711057
2024-07-21 03:02:10,824 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,825 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-07-21 03:02:10,826 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,827 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-07-20
2024-07-21 03:02:10,866 INFO [collect_river_data] running in process 711067
2024-07-21 03:02:10,867 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,868 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-07-21 03:02:10,869 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,869 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-07-20
2024-07-21 03:02:10,875 INFO [collect_river_data] running in process 711065
2024-07-21 03:02:10,876 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,876 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-07-21 03:02:10,877 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,878 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-07-20
2024-07-21 03:02:10,888 INFO [collect_river_data] running in process 711068
2024-07-21 03:02:10,890 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,893 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-07-21 03:02:10,900 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,900 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-07-20
2024-07-21 03:02:10,901 INFO [collect_river_data] running in process 711066
2024-07-21 03:02:10,902 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,903 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-07-21 03:02:10,904 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,905 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-07-20
2024-07-21 03:02:10,919 INFO [collect_river_data] running in process 711069
2024-07-21 03:02:10,921 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:10,923 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-07-21 03:02:10,924 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:10,925 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-07-20
2024-07-21 03:02:11,014 INFO [collect_NeahBay_ssh] 2024-07-21T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-07-21 03:02:11,017 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-21 03:02:11,155 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-07-20 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-07-21 03:02:11,156 INFO [collect_river_data] ECCC Englishman river data collection for 2024-07-20 completed
2024-07-21 03:02:11,354 INFO [get_vfpa_hadcp] running in process 711074
2024-07-21 03:02:11,354 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:11,355 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5590
2024-07-21 03:02:11,356 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:11,357 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-20
2024-07-21 03:02:11,362 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-07-20 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-07-21 03:02:11,363 INFO [collect_river_data] ECCC Fraser river data collection for 2024-07-20 completed
2024-07-21 03:02:11,412 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-07-21 03:02:11,413 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-07-20 completed
2024-07-21 03:02:11,492 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-07-21 03:02:11,493 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-07-20 completed
2024-07-21 03:02:11,525 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-07-21 03:02:11,525 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-07-20 completed
2024-07-21 03:02:11,529 INFO [crop_gribs] running in process 711077
2024-07-21 03:02:11,530 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:11,530 INFO [crop_gribs] publishing log messages to tcp://*:5591
2024-07-21 03:02:11,532 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:11,532 INFO [crop_gribs] cropping 2024-07-21 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-07-21 03:02:11,554 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-07-21 03:02:11,555 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-07-20 completed
2024-07-21 03:02:11,570 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240721/12/
2024-07-21 03:02:11,581 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 03:02:11,621 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-07-21 03:02:11,623 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-07-20 completed
2024-07-21 03:02:11,627 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:11,650 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-07-20 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-07-21 03:02:11,651 INFO [collect_river_data] ECCC Capilano river data collection for 2024-07-20 completed
2024-07-21 03:02:11,654 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-07-21 03:02:11,654 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-07-20 completed
2024-07-21 03:02:11,664 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-07-21 03:02:11,665 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-07-20 completed
2024-07-21 03:02:11,705 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-07-21 03:02:11,706 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-07-20 completed
2024-07-21 03:02:11,709 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-07-21 03:02:11,710 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-07-21 03:02:11,711 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-07-20 completed
2024-07-21 03:02:11,712 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-07-20 completed
2024-07-21 03:02:11,727 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-07-21 03:02:11,727 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-07-20 completed
2024-07-21 03:02:11,852 INFO [get_onc_ctd] running in process 711071
2024-07-21 03:02:11,853 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:11,853 INFO [get_onc_ctd] publishing log messages to tcp://*:5564
2024-07-21 03:02:11,854 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:11,855 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-07-20
2024-07-21 03:02:11,925 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:11,937 INFO [get_onc_ctd] running in process 711072
2024-07-21 03:02:11,938 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:11,939 INFO [get_onc_ctd] publishing log messages to tcp://*:5592
2024-07-21 03:02:11,940 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:11,940 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-07-20
2024-07-21 03:02:11,945 INFO [grib_to_netcdf] running in process 711236
2024-07-21 03:02:11,946 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:11,946 INFO [grib_to_netcdf] publishing log messages to tcp://*:5593
2024-07-21 03:02:11,947 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:11,948 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-21 forecast2 run
2024-07-21 03:02:11,954 INFO [get_onc_ferry] running in process 711073
2024-07-21 03:02:11,955 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:11,956 INFO [get_onc_ferry] publishing log messages to tcp://*:5594
2024-07-21 03:02:11,957 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:11,958 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-07-20
2024-07-21 03:02:12,132 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:12,324 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:12,522 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:12,752 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:12,948 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:13,144 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:13,362 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:13,404 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-20 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202407.nc
2024-07-21 03:02:13,405 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-07 netcdf file
2024-07-21 03:02:13,557 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:13,756 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:13,960 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:14,092 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-07-20
2024-07-21 03:02:14,190 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:14,383 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 03:02:14,465 INFO [make_ssh_files] running in process 711502
2024-07-21 03:02:14,466 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:14,467 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2024-07-21 03:02:14,468 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:14,469 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-07-21 NOAA Neah Bay 00Z observation and forecast values
2024-07-21 03:02:14,573 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2024-07-21 03:02:14,633 CRITICAL [get_onc_ctd] 2024-07-20 ONC SCVIP CTD T&S file creation failed
2024-07-21 03:02:15,025 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2024-07-21 03:02:16,166 INFO [ping_erddap] running in process 711554
2024-07-21 03:02:16,167 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:16,168 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-21 03:02:16,169 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:16,171 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2024-07-21 03:02:16,172 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-21 03:02:19,325 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d16.nc
2024-07-21 03:02:19,522 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d17.nc
2024-07-21 03:02:19,704 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc
2024-07-21 03:02:19,887 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc
2024-07-21 03:02:20,077 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-21 03:02:20,260 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d21.nc
2024-07-21 03:02:20,404 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d22.nc
2024-07-21 03:02:20,441 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d22.nc
2024-07-21 03:02:20,622 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d23.nc
2024-07-21 03:02:20,795 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d24.nc
2024-07-21 03:02:20,909 INFO [make_ssh_files] sea surface height boundary file for 2024-07-21 forecast2 run created
2024-07-21 03:02:20,912 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-21 03:02:21,118 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-21 03:02:21,121 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-21 03:02:22,474 INFO [make_runoff_file] running in process 711586
2024-07-21 03:02:22,475 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:22,476 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-07-21 03:02:22,477 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:22,502 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-21 03:02:22,570 INFO [make_runoff_file] 2024-07-21 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-21 03:02:22,572 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-21 03:02:22,794 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d23.nc
2024-07-21 03:02:22,801 INFO [grib_to_netcdf] 2024-07-21 NEMO-atmos forcing file for forecast2 created
2024-07-21 03:02:22,803 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-21 03:02:23,102 INFO [make_v202111_runoff_file] running in process 711584
2024-07-21 03:02:23,103 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:23,104 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564
2024-07-21 03:02:23,105 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:23,106 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-20
2024-07-21 03:02:24,023 ERROR [make_v202111_runoff_file] no 2024-07-20 discharge obs for Skagit_MountVernon: patching
2024-07-21 03:02:24,507 ERROR [make_v202111_runoff_file] no 2024-07-20 discharge obs for Snohomish_Monroe: patching
2024-07-21 03:02:24,703 ERROR [make_v202111_runoff_file] no 2024-07-20 discharge obs for Nisqually_McKenna: patching
2024-07-21 03:02:24,788 ERROR [make_v202111_runoff_file] no 2024-07-20 discharge obs for Greenwater_Greenwater: patching
2024-07-21 03:02:26,072 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-20: /results/forcing/rivers/R202108Dailies_y2024m07d20.nc
2024-07-21 03:02:26,073 INFO [make_v202111_runoff_file] 2024-07-20 runoff file creation completed
2024-07-21 03:02:26,075 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-21 03:02:26,280 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-07-21 03:02:26,285 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-07-21 03:02:26,291 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-07-21 03:02:26,295 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-07-21 03:02:27,528 INFO [upload_forcing] running in process 711625
2024-07-21 03:02:27,529 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:27,530 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-07-21 03:02:27,531 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:27,541 INFO [upload_forcing] running in process 711628
2024-07-21 03:02:27,542 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:27,543 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-07-21 03:02:27,544 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:27,546 INFO [upload_forcing] running in process 711627
2024-07-21 03:02:27,547 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:27,548 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-21 03:02:27,549 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:27,567 INFO [upload_forcing] running in process 711626
2024-07-21 03:02:27,568 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:27,569 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-21 03:02:27,570 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:28,937 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d20.nc
2024-07-21 03:02:28,965 INFO [upload_forcing] forecast2 2024-07-21 forcing files upload to optimum-hindcast completed
2024-07-21 03:02:28,969 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 03:02:29,097 INFO [upload_forcing] forecast2 2024-07-21 forcing files upload to orcinus-nowcast-agrif completed
2024-07-21 03:02:29,195 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 03:02:30,408 INFO [upload_forcing] forecast2 2024-07-21 forcing files upload to arbutus.cloud-nowcast completed
2024-07-21 03:02:30,412 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 03:02:30,619 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 03:02:31,354 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-07-20
2024-07-21 03:02:31,704 WARNING [get_onc_ctd] no SEVIP salinity data
2024-07-21 03:02:31,840 INFO [make_forcing_links] running in process 711656
2024-07-21 03:02:31,841 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:31,842 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-21 03:02:31,843 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:31,940 INFO [get_onc_ctd] 2024-07-20 ONC SEVIP CTD T&S file created
2024-07-21 03:02:31,942 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-07-21 03:02:32,159 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-07-21 03:02:32,407 WARNING [get_onc_ferry] No ONC TWDP TSG temperature,conductivity,salinity data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:32,412 WARNING [get_onc_ferry] No ONC TWDP TSG temperature data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:32,413 WARNING [get_onc_ferry] No ONC TWDP TSG conductivity data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:32,415 WARNING [get_onc_ferry] No ONC TWDP TSG salinity data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:32,416 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-07-20
2024-07-21 03:02:33,211 INFO [make_forcing_links] forecast2 2024-07-21 forcing file links on arbutus.cloud-nowcast created
2024-07-21 03:02:33,216 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-21 03:02:33,316 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation,oxygen_corrected,temperature data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:33,322 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:33,324 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:33,326 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR temperature data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:33,327 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-07-20
2024-07-21 03:02:33,372 INFO [ping_erddap] running in process 711663
2024-07-21 03:02:33,374 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:33,375 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-21 03:02:33,377 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:33,379 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-07-21 03:02:33,428 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 03:02:33,432 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-21 03:02:34,147 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence,chlorophyll,turbidity data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:34,153 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:34,155 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL chlorophyll data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:34,157 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL turbidity data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:34,158 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-07-20
2024-07-21 03:02:34,879 INFO [upload_forcing] forecast2 2024-07-21 forcing files upload to robot.graham completed
2024-07-21 03:02:34,884 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 03:02:35,474 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:35,479 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:35,481 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2024-07-20; substituting empty dataset
2024-07-21 03:02:35,482 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-07-20
2024-07-21 03:02:35,565 INFO [run_NEMO] running in process 29419
2024-07-21 03:02:35,567 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:35,568 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-21 03:02:35,869 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:36,823 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-21 03:02:36,827 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/20jul24forecast2_2024-07-21T030235.970476-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-21 03:02:36,864 INFO [run_NEMO] forecast2 NEMO run for 2024-07-21 on arbutus.cloud-nowcast started
2024-07-21 03:02:37,026 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-21 03:02:39,066 INFO [watch_NEMO] running in process 29671
2024-07-21 03:02:39,068 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:02:39,069 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-21 03:02:39,070 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:02:39,157 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-21 03:02:52,414 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-07-20
2024-07-21 03:03:01,001 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-07-20
2024-07-21 03:03:09,386 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-07-20; substituting empty dataset
2024-07-21 03:03:09,387 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-07-20
2024-07-21 03:03:17,336 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-20; substituting empty dataset
2024-07-21 03:03:17,338 WARNING [get_onc_ferry] ONC TWDP TSG temperature data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,359 WARNING [get_onc_ferry] ONC TWDP TSG conductivity data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,421 WARNING [get_onc_ferry] ONC TWDP TSG salinity data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,439 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,456 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,473 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR temperature data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,491 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,508 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL chlorophyll data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,526 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL turbidity data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,543 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,560 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,664 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:03:17,681 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-21 03:04:31,814 INFO [get_onc_ferry] 2024-07-20 ONC TWDP ferry data file created
2024-07-21 03:04:31,817 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-07-21 03:04:32,025 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-07-21 03:04:33,325 INFO [ping_erddap] running in process 711681
2024-07-21 03:04:33,326 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:04:33,327 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-21 03:04:33,328 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:04:33,330 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-07-21 03:04:33,332 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-21 03:07:39,148 CRITICAL [watch_NEMO] arbutus.cloud-nowcast forecast2/20jul24 run aborted: /nemoShare/MEOPAR/SalishSea/forecast2/20jul24/output.abort.nc
2024-07-21 03:07:39,917 CRITICAL [watch_NEMO] arbutus.cloud-nowcast forecast2/20jul24 run failed: final time step is 17800561 not 17803260
2024-07-21 03:07:39,944 CRITICAL [watch_NEMO] arbutus.cloud-nowcast forecast2/20jul24 run failed; no physics restart file: /nemoShare/MEOPAR/SalishSea/forecast2/20jul24/SalishSea_17802720_restart.nc
2024-07-21 03:07:39,973 CRITICAL [watch_NEMO] arbutus.cloud-nowcast forecast2/20jul24 run failed; 1 or more E R R O R in: /nemoShare/MEOPAR/SalishSea/forecast2/20jul24/ocean.output
2024-07-21 03:07:40,001 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-07-21 03:07:40,003 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-21 03:07:40,363 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 03:07:40,371 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 03:07:40,375 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-20'], host='localhost')
2024-07-21 03:07:41,795 INFO [download_results] running in process 711705
2024-07-21 03:07:41,797 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:07:41,798 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 03:07:41,799 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:07:41,801 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/20jul24 to /results/SalishSea/forecast2.202111
2024-07-21 03:07:44,247 INFO [make_ww3_current_file] running in process 29843
2024-07-21 03:07:44,248 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:07:44,249 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-21 03:07:44,250 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:07:44,251 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-21 forecast2 run
2024-07-21 03:07:44,287 INFO [make_ww3_wind_file] running in process 29841
2024-07-21 03:07:44,288 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:07:44,289 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-21 03:07:44,291 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:07:44,292 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-21 forecast2 run
2024-07-21 03:07:47,122 INFO [download_results] forecast2 2024-07-20 results files from arbutus.cloud-nowcast downloaded
2024-07-21 03:07:47,126 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 03:07:47,332 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-07-20'], host='localhost')
2024-07-21 03:07:49,345 INFO [make_CHS_currents_file] running in process 711711
2024-07-21 03:07:49,347 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:07:49,348 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-21 03:07:49,349 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:07:49,655 CRITICAL [make_CHS_currents_file] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('/results/SalishSea/forecast2.202111/20jul24/SalishSea_1h_20240722_20240723_grid_U.nc',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '14f317b1-671c-4df5-8d53-5d0170c5ef89']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work
    checklist = self.worker_func(
                ^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_CHS_currents_file.py", line 121, in make_CHS_currents_file
    urot5, vrot5, urot10, vrot10 = _read_avg_unstagger_rotate(
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_CHS_currents_file.py", line 155, in _read_avg_unstagger_rotate
    uds = xarray.open_dataset(src_dir / ufile)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/api.py", line 571, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 645, in open_dataset
    store = NetCDF4DataStore.open(
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 408, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 355, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 417, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/netCDF4_.py", line 411, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/SalishSeaCast/nowcast-env/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 2469, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
FileNotFoundError: [Errno 2] No such file or directory: '/results/SalishSea/forecast2.202111/20jul24/SalishSea_1h_20240722_20240723_grid_U.nc'
2024-07-21 03:07:54,405 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-21 forecast2 run
2024-07-21 03:07:54,407 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-21 03:09:02,034 CRITICAL [make_ww3_current_file] unhandled exception:
Traceback (most recent call last):
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'h5netcdf.core.File'>, ('/nemoShare/MEOPAR/SalishSea/forecast2/20jul24/SalishSea_1h_20240722_20240723_grid_U.nc',), 'r', (('decode_vlen_strings', True), ('driver', None), ('invalid_netcdf', None)), 'c5acda2d-93a9-4ba6-b025-c02fb338021e']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/nemoShare/MEOPAR/nowcast-sys/NEMO_Nowcast/nemo_nowcast/worker.py", line 391, in _do_work
    checklist = self.worker_func(
                ^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/nowcast/workers/make_ww3_current_file.py", line 146, in make_ww3_current_file
    with xarray.open_mfdataset(
         ^^^^^^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/api.py", line 1054, in open_mfdataset
    datasets = [open_(p, **open_kwargs) for p in paths]
                ^^^^^^^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/api.py", line 571, in open_dataset
    backend_ds = backend.open_dataset(
                 ^^^^^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py", line 405, in open_dataset
    store = H5NetCDFStore.open(
            ^^^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py", line 185, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py", line 131, in __init__
    self._filename = find_root_and_group(self.ds)[0].filename
                                         ^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py", line 196, in ds
    return self._acquire()
           ^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/h5netcdf_.py", line 188, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/h5netcdf/core.py", line 1054, in __init__
    self._h5file = self._h5py.File(
                   ^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/h5py/_hl/files.py", line 562, in __init__
    fid = make_fid(name, mode, userblock_size, fapl, fcpl, swmr=swmr)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/nemoShare/MEOPAR/nowcast-sys/nowcast-env/lib/python3.12/site-packages/h5py/_hl/files.py", line 235, in make_fid
    fid = h5f.open(name, flags, fapl=fapl)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5f.pyx", line 102, in h5py.h5f.open
FileNotFoundError: [Errno 2] Unable to synchronously open file (unable to open file: name = '/nemoShare/MEOPAR/SalishSea/forecast2/20jul24/SalishSea_1h_20240722_20240723_grid_U.nc', errno = 2, error message = 'No such file or directory', flags = 0, o_flags = 0)
2024-07-21 03:37:19,197 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 03:37:19,555 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-05-26'], host='localhost')
2024-07-21 03:37:19,563 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 03:37:19,566 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 03:37:20,792 INFO [download_results] running in process 711823
2024-07-21 03:37:20,793 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:37:20,794 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-21 03:37:20,796 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:37:20,797 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26may18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 03:37:24,846 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 03:38:05,754 INFO [download_results] hindcast 2018-05-26 results files from optimum-hindcast downloaded
2024-07-21 03:38:05,763 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 03:38:05,972 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-05-26'], host='localhost')
2024-07-21 03:38:07,321 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 03:38:07,526 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-may', 'robot.graham'], host='localhost')
2024-07-21 03:38:08,809 INFO [archive_tarball] running in process 711869
2024-07-21 03:38:08,811 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 03:38:08,812 INFO [archive_tarball] publishing log messages to tcp://*:5565
2024-07-21 03:38:08,813 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-21 03:38:08,814 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-may18.tar from /ocean/sallen/timbucktwo/oxygen/*may18/
2024-07-21 03:43:52,162 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-may18.index from /ocean/dlatorne/oxygen-may18.tar
2024-07-21 03:43:52,247 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-may18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-21 03:52:56,883 INFO [archive_tarball] hindcast *may18 results files archived to robot.graham
2024-07-21 03:52:56,887 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-21 05:02:29,399 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 05:02:29,766 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-06-01'], host='localhost')
2024-07-21 05:02:29,773 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 05:02:29,777 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 05:02:31,003 INFO [download_results] running in process 712138
2024-07-21 05:02:31,004 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 05:02:31,006 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-21 05:02:31,007 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 05:02:31,008 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jun18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 05:02:35,265 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 05:03:10,993 INFO [download_results] hindcast 2018-06-01 results files from optimum-hindcast downloaded
2024-07-21 05:03:11,005 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 05:03:11,208 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-06-01'], host='localhost')
2024-07-21 05:03:12,402 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 06:32:40,029 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 06:32:40,383 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-06-06'], host='localhost')
2024-07-21 06:32:40,390 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 06:32:40,395 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 06:32:41,632 INFO [download_results] running in process 712645
2024-07-21 06:32:41,634 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 06:32:41,635 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-21 06:32:41,636 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 06:32:41,637 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06jun18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 06:32:45,809 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 06:33:21,432 INFO [download_results] hindcast 2018-06-06 results files from optimum-hindcast downloaded
2024-07-21 06:33:21,443 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 06:33:21,646 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-06-06'], host='localhost')
2024-07-21 06:33:22,798 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 07:57:50,322 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 07:57:50,684 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-06-11'], host='localhost')
2024-07-21 07:57:50,692 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 07:57:50,696 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 07:57:51,942 INFO [download_results] running in process 713071
2024-07-21 07:57:51,943 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 07:57:51,945 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 07:57:51,946 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 07:57:51,947 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11jun18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 07:57:56,162 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 07:58:31,787 INFO [download_results] hindcast 2018-06-11 results files from optimum-hindcast downloaded
2024-07-21 07:58:31,796 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 07:58:32,005 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-06-11'], host='localhost')
2024-07-21 07:58:33,172 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 09:03:16,432 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240721/12/
2024-07-21 09:03:16,433 INFO [collect_weather] 2024-07-21 2.5km weather forecast 12 collection complete
2024-07-21 09:03:16,436 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-21 09:03:16,624 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240721/12/
2024-07-21 09:03:16,625 INFO [crop_gribs] 2024-07-21 12 GRIBs cropping complete
2024-07-21 09:03:16,751 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 09:03:16,757 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 09:03:16,762 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 09:03:16,768 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-07-20'], host='localhost')
2024-07-21 09:03:16,773 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-07-21 09:03:16,777 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-07-21 09:03:16,782 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-07-21 09:03:16,786 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-07-21 09:03:16,790 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-07-21 09:03:16,799 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-21 09:03:17,388 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-07-21 09:03:18,470 INFO [collect_weather] running in process 713794
2024-07-21 09:03:18,472 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:18,473 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-21 09:03:18,474 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:18,483 INFO [collect_NeahBay_ssh] running in process 713791
2024-07-21 09:03:18,483 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:18,484 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-07-21 09:03:18,485 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:18,486 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-07-21
2024-07-21 09:03:18,702 INFO [download_live_ocean] running in process 713793
2024-07-21 09:03:18,703 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:18,703 INFO [download_live_ocean] publishing log messages to tcp://*:5566
2024-07-21 09:03:18,704 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:18,705 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-07-21
2024-07-21 09:03:18,729 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-07-21 09:03:19,007 INFO [collect_river_data] running in process 713785
2024-07-21 09:03:19,008 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:19,009 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-21 09:03:19,010 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:19,011 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-07-20
2024-07-21 09:03:19,051 INFO [collect_river_data] running in process 713789
2024-07-21 09:03:19,052 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:19,053 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-21 09:03:19,054 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:19,055 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-07-20
2024-07-21 09:03:19,110 INFO [collect_river_data] running in process 713788
2024-07-21 09:03:19,111 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:19,112 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-21 09:03:19,113 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:19,114 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-07-20
2024-07-21 09:03:19,166 INFO [make_turbidity_file] running in process 713790
2024-07-21 09:03:19,167 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:19,168 INFO [make_turbidity_file] publishing log messages to tcp://*:5570
2024-07-21 09:03:19,169 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:19,170 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-07-21
2024-07-21 09:03:19,179 INFO [collect_river_data] running in process 713787
2024-07-21 09:03:19,180 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:19,181 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-21 09:03:19,182 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:19,183 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-07-20
2024-07-21 09:03:19,228 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-07-20&endDT=2024-07-20 "HTTP/1.1 200 OK"
2024-07-21 09:03:19,234 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-07-21 09:03:19,235 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-07-20 completed
2024-07-21 09:03:19,236 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 09:03:19,335 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-07-20&endDT=2024-07-20 "HTTP/1.1 200 OK"
2024-07-21 09:03:19,341 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-07-21 09:03:19,342 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-07-20 completed
2024-07-21 09:03:19,375 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-07-20&endDT=2024-07-20 "HTTP/1.1 200 OK"
2024-07-21 09:03:19,380 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-07-21 09:03:19,382 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-07-20 completed
2024-07-21 09:03:19,421 INFO [download_live_ocean] downloaded 10125193 bytes from https://liveocean.apl.uw.edu/output/f2024.07.21/ubc.nc to /results/forcing/LiveOcean/downloaded/20240721/low_passed_UBC.nc
2024-07-21 09:03:19,422 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-07-21 09:03:19,433 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 09:03:19,460 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-07-20&endDT=2024-07-20 "HTTP/1.1 200 OK"
2024-07-21 09:03:19,465 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-07-20 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-07-21 09:03:19,466 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-07-20 completed
2024-07-21 09:03:19,534 INFO [crop_gribs] running in process 713795
2024-07-21 09:03:19,535 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:19,536 INFO [crop_gribs] publishing log messages to tcp://*:5572
2024-07-21 09:03:19,537 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:19,538 INFO [crop_gribs] cropping 2024-07-21 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-07-21 09:03:19,556 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240721/18/
2024-07-21 09:03:19,589 INFO [grib_to_netcdf] running in process 713822
2024-07-21 09:03:19,590 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:19,591 INFO [grib_to_netcdf] publishing log messages to tcp://*:5573
2024-07-21 09:03:19,592 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:19,593 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-21 nowcast and forecast runs
2024-07-21 09:03:19,619 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 09:03:19,815 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-21 09:03:20,308 INFO [collect_NeahBay_ssh] 2024-07-21T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-07-21 09:03:20,310 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-21 09:03:20,462 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240721/low_passed_UBC.nc
2024-07-21 09:03:20,462 INFO [download_live_ocean] 2024-07-21 Live Ocean file for Salish Sea western boundary downloaded
2024-07-21 09:03:20,491 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 09:03:20,496 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-07-21 09:03:20,892 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-07-21'], host='localhost')
2024-07-21 09:03:22,158 INFO [make_turbidity_file] 2024-07-21 Fraser River turbidity file creation complete
2024-07-21 09:03:22,160 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-07-21 09:03:22,939 INFO [make_ssh_files] running in process 713967
2024-07-21 09:03:22,940 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:22,941 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2024-07-21 09:03:22,943 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:22,944 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-07-21 NOAA Neah Bay 06Z observation and forecast values
2024-07-21 09:03:22,945 INFO [make_live_ocean_files] running in process 713981
2024-07-21 09:03:22,946 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:22,947 INFO [make_live_ocean_files] publishing log messages to tcp://*:5566
2024-07-21 09:03:22,948 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:22,979 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-07-21 Live Ocean run
2024-07-21 09:03:27,532 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d17.nc
2024-07-21 09:03:27,700 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc
2024-07-21 09:03:27,862 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc
2024-07-21 09:03:28,028 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m07d21.nc
2024-07-21 09:03:28,032 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-21 09:03:28,222 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d21.nc
2024-07-21 09:03:28,391 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d22.nc
2024-07-21 09:03:28,563 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d23.nc
2024-07-21 09:03:28,734 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d24.nc
2024-07-21 09:03:28,832 INFO [make_ssh_files] sea surface height boundary file for 2024-07-21 nowcast run created
2024-07-21 09:03:28,836 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-21 09:03:29,047 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-21 09:03:29,052 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-21 09:03:30,331 INFO [make_runoff_file] running in process 714036
2024-07-21 09:03:30,333 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:30,334 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-07-21 09:03:30,335 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:30,372 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-21 09:03:30,445 INFO [make_runoff_file] 2024-07-21 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-21 09:03:30,447 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-21 09:03:30,949 INFO [make_v202111_runoff_file] running in process 714034
2024-07-21 09:03:30,950 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:03:30,951 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564
2024-07-21 09:03:30,952 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:03:30,953 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-20
2024-07-21 09:03:33,447 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-20: /results/forcing/rivers/R202108Dailies_y2024m07d20.nc
2024-07-21 09:03:33,448 INFO [make_v202111_runoff_file] 2024-07-20 runoff file creation completed
2024-07-21 09:03:33,450 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-21 09:03:35,156 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d22.nc
2024-07-21 09:03:38,888 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d23.nc
2024-07-21 09:03:38,906 INFO [grib_to_netcdf] 2024-07-21 NEMO-atmos forcing file for nowcast+ created
2024-07-21 09:03:38,910 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-21 09:04:02,460 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d21.nc
2024-07-21 09:04:02,461 INFO [make_live_ocean_files] 2024-07-21 Live Ocean western boundary conditions files created
2024-07-21 09:04:02,463 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-07-21 09:04:02,677 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-07-21 09:04:02,682 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-07-21 09:04:02,686 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-07-21 09:04:02,691 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-07-21 09:04:02,696 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-07-21 09:04:03,833 INFO [ping_erddap] running in process 714077
2024-07-21 09:04:03,834 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:04:03,835 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-21 09:04:03,837 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:04:03,839 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-07-21 09:04:03,841 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-21 09:04:03,869 INFO [upload_forcing] running in process 714079
2024-07-21 09:04:03,870 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:04:03,872 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-21 09:04:03,873 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:04:03,912 INFO [upload_forcing] running in process 714081
2024-07-21 09:04:03,913 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:04:03,914 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-21 09:04:03,915 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:04:03,928 INFO [upload_forcing] running in process 714080
2024-07-21 09:04:03,929 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:04:03,930 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-21 09:04:03,931 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:04:03,935 INFO [upload_forcing] running in process 714082
2024-07-21 09:04:03,936 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:04:03,937 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-21 09:04:03,938 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:04:06,287 INFO [upload_forcing] nowcast+ 2024-07-21 forcing files upload to optimum-hindcast completed
2024-07-21 09:04:06,290 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 09:04:06,324 INFO [upload_forcing] nowcast+ 2024-07-21 forcing files upload to orcinus-nowcast-agrif completed
2024-07-21 09:04:06,515 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 09:04:14,155 INFO [upload_forcing] nowcast+ 2024-07-21 forcing files upload to robot.graham completed
2024-07-21 09:04:14,160 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 09:04:16,074 INFO [upload_forcing] nowcast+ 2024-07-21 forcing files upload to arbutus.cloud-nowcast completed
2024-07-21 09:04:16,079 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 09:04:16,283 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 09:04:17,417 INFO [make_forcing_links] running in process 714150
2024-07-21 09:04:17,419 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:04:17,420 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-21 09:04:17,421 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:04:18,720 INFO [make_forcing_links] nowcast+ 2024-07-21 forcing file links on arbutus.cloud-nowcast created
2024-07-21 09:04:18,724 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-21 09:04:18,927 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 09:04:26,037 INFO [run_NEMO] running in process 992
2024-07-21 09:04:26,038 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:04:26,040 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-21 09:04:26,161 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:04:26,944 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/21jul24nowcast_2024-07-21T090426.090724-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-21 09:04:26,964 INFO [run_NEMO] nowcast NEMO run for 2024-07-21 on arbutus.cloud-nowcast started
2024-07-21 09:04:26,969 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-21 09:04:27,168 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-21 09:04:29,150 INFO [watch_NEMO] running in process 1255
2024-07-21 09:04:29,152 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:04:29,153 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-21 09:04:29,155 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:04:29,239 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-21 09:09:29,234 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17798851 = 2024-07-21 05:00:00 UTC, 20.8% complete
2024-07-21 09:14:29,235 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17799334 = 2024-07-21 10:22:00 UTC, 43.2% complete
2024-07-21 09:19:29,236 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17799843 = 2024-07-21 16:01:20 UTC, 66.8% complete
2024-07-21 09:23:00,203 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 09:23:00,550 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-06-16'], host='localhost')
2024-07-21 09:23:00,556 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 09:23:00,562 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 09:23:01,762 INFO [download_results] running in process 714309
2024-07-21 09:23:01,763 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:23:01,764 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 09:23:01,765 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:23:01,766 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16jun18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 09:23:06,020 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 09:23:41,705 INFO [download_results] hindcast 2018-06-16 results files from optimum-hindcast downloaded
2024-07-21 09:23:41,716 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 09:23:41,925 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-06-16'], host='localhost')
2024-07-21 09:23:43,104 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 09:24:29,234 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17800323 = 2024-07-21 21:21:20 UTC, 89.0% complete
2024-07-21 09:29:29,234 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-07-21 09:29:29,238 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-21 09:29:29,598 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 09:29:29,605 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 09:29:30,836 INFO [download_results] running in process 714374
2024-07-21 09:29:30,839 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:29:30,840 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 09:29:30,841 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:29:30,842 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/21jul24 to /results/SalishSea/nowcast-blue.202111
2024-07-21 09:29:30,844 INFO [make_forcing_links] running in process 714373
2024-07-21 09:29:30,845 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:29:30,846 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-21 09:29:30,848 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:29:31,743 INFO [make_forcing_links] ssh 2024-07-21 forcing file links on arbutus.cloud-nowcast created
2024-07-21 09:29:31,747 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-21 09:29:31,976 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 09:29:35,355 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/21jul24forecast_2024-07-21T092934.424288-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-21 09:29:35,375 INFO [run_NEMO] forecast NEMO run for 2024-07-21 on arbutus.cloud-nowcast started
2024-07-21 09:29:35,380 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-21 09:29:35,587 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-21 09:34:39,167 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17801002 = 2024-07-22 04:54:00 UTC, 13.6% complete
2024-07-21 09:39:38,829 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17801441 = 2024-07-22 09:46:40 UTC, 27.2% complete
2024-07-21 09:44:39,167 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17801939 = 2024-07-22 15:18:40 UTC, 42.5% complete
2024-07-21 09:49:39,098 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17802374 = 2024-07-22 20:08:40 UTC, 56.0% complete
2024-07-21 09:49:50,770 INFO [download_results] nowcast 2024-07-21 results files from arbutus.cloud-nowcast downloaded
2024-07-21 09:49:50,775 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 09:49:50,983 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 09:49:50,988 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-07-20'], host='localhost')
2024-07-21 09:49:50,992 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 09:49:53,005 INFO [make_CHS_currents_file] running in process 714491
2024-07-21 09:49:53,006 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:49:53,007 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-21 09:49:53,008 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:49:54,600 INFO [make_plots] running in process 714488
2024-07-21 09:49:54,601 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:49:54,602 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-21 09:49:54,603 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:49:54,605 INFO [make_plots] running in process 714490
2024-07-21 09:49:54,606 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 09:49:54,607 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-21 09:49:54,613 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-21 09:49:55,245 INFO [make_plots] preparing render list for 2024-07-21 NEMO nowcast-blue research figures
2024-07-21 09:49:55,246 INFO [make_plots] preparing render list for 2024-07-20 NEMO nowcast-blue comparison figures
2024-07-21 09:49:56,313 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-blue research render list
2024-07-21 09:49:56,998 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-blue research render list
2024-07-21 09:49:56,999 INFO [make_plots] added Currents_sections_and_surface figure to 2024-07-21 NEMO nowcast-blue research render list
2024-07-21 09:49:57,000 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-07-21 NEMO nowcast-blue research render list
2024-07-21 09:49:57,001 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-07-21 NEMO nowcast-blue research render list
2024-07-21 09:49:57,002 INFO [make_plots] starting to render nemo nowcast research 21jul24 figures
2024-07-21 09:50:14,643 INFO [make_plots] added SH_wind figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,645 INFO [make_plots] added Compare_VENUS_East figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,646 INFO [make_plots] added Compare_VENUS_Central figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,647 INFO [make_plots] added Central_ADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,647 INFO [make_plots] added Central_depavADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,648 INFO [make_plots] added Central_timeavADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,649 INFO [make_plots] added East_ADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,650 INFO [make_plots] added East_depavADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,651 INFO [make_plots] added East_timeavADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,652 INFO [make_plots] added ddl_ADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,653 INFO [make_plots] added ddl_depavADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,654 INFO [make_plots] added ddl_timeavADCP figure to 2024-07-20 NEMO nowcast-blue comparison render list
2024-07-21 09:50:14,654 INFO [make_plots] starting to render nemo nowcast comparison 20jul24 figures
2024-07-21 09:50:22,988 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-21 for nowcast
2024-07-21 09:50:22,992 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-21 09:51:06,281 INFO [make_plots] finished rendering nemo nowcast research 21jul24 figures
2024-07-21 09:51:06,284 INFO [make_plots] nemo research plots for 2024-07-21 nowcast completed
2024-07-21 09:51:06,295 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-21 09:51:33,833 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available
2024-07-21 09:51:34,392 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-21 09:51:34,892 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-21 09:51:35,389 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-21 09:51:35,894 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-21 09:51:36,393 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-21 09:51:36,893 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-21 09:51:37,393 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-21 09:51:37,892 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-21 09:51:38,395 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-21 09:51:38,396 INFO [make_plots] finished rendering nemo nowcast comparison 20jul24 figures
2024-07-21 09:51:38,495 INFO [make_plots] nemo comparison plots for 2024-07-20 nowcast completed
2024-07-21 09:51:38,498 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-21 09:54:38,502 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17802824 = 2024-07-23 01:08:40 UTC, 69.9% complete
2024-07-21 09:59:38,502 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17803315 = 2024-07-23 06:36:00 UTC, 85.0% complete
2024-07-21 10:04:38,501 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17803780 = 2024-07-23 11:46:00 UTC, 99.4% complete
2024-07-21 10:09:38,505 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-07-21 10:09:38,509 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-21 10:09:38,989 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 10:09:38,997 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 10:09:39,001 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:09:39,007 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:09:39,011 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:09:39,015 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:09:39,021 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:09:40,268 INFO [download_results] running in process 714782
2024-07-21 10:09:40,269 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:40,270 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 10:09:40,271 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:40,273 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/21jul24 to /results/SalishSea/forecast.202111
2024-07-21 10:09:40,274 INFO [upload_forcing] running in process 714779
2024-07-21 10:09:40,275 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:40,276 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-07-21 10:09:40,277 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:40,278 INFO [upload_forcing] running in process 714781
2024-07-21 10:09:40,279 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:40,280 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-21 10:09:40,281 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:40,284 INFO [upload_forcing] running in process 714778
2024-07-21 10:09:40,285 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:40,287 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-21 10:09:40,287 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:40,301 INFO [upload_forcing] running in process 714780
2024-07-21 10:09:40,302 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:40,303 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-21 10:09:40,304 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:40,904 INFO [upload_forcing] turbidity 2024-07-21 forcing files upload to optimum-hindcast completed
2024-07-21 10:09:40,908 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 10:09:41,178 INFO [upload_forcing] turbidity 2024-07-21 forcing files upload to orcinus-nowcast-agrif completed
2024-07-21 10:09:41,182 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 10:09:41,314 INFO [upload_forcing] turbidity 2024-07-21 forcing files upload to arbutus.cloud-nowcast completed
2024-07-21 10:09:41,382 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:09:41,389 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 10:09:41,580 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:09:42,079 INFO [make_ww3_wind_file] running in process 2640
2024-07-21 10:09:42,080 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:42,083 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-21 10:09:42,094 INFO [make_ww3_current_file] running in process 2638
2024-07-21 10:09:42,095 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:42,096 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-21 10:09:42,111 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:42,113 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-21 forecast run
2024-07-21 10:09:42,118 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:42,119 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-21 forecast run
2024-07-21 10:09:42,593 INFO [make_forcing_links] running in process 714814
2024-07-21 10:09:42,595 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:42,596 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-21 10:09:42,598 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:42,815 INFO [make_forcing_links] running in process 714816
2024-07-21 10:09:42,816 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:42,817 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2024-07-21 10:09:42,818 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:43,457 INFO [make_forcing_links] nowcast-agrif 2024-07-21 forcing file links on orcinus-nowcast-agrif created
2024-07-21 10:09:43,461 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-21 10:09:43,679 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:09:44,987 INFO [run_NEMO_agrif] running in process 714830
2024-07-21 10:09:44,989 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:44,990 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5565
2024-07-21 10:09:44,991 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:45,962 INFO [upload_forcing] turbidity 2024-07-21 forcing files upload to robot.graham completed
2024-07-21 10:09:45,967 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-21 10:09:47,627 INFO [make_forcing_links] nowcast-green 2024-07-21 forcing file links on arbutus.cloud-nowcast created
2024-07-21 10:09:47,631 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-21 10:09:47,847 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 10:09:51,058 INFO [run_NEMO_agrif] job id for 21jul24nowcast-agrif: 11101963.orca2.ibb
2024-07-21 10:09:51,059 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-07-21 queued on orcinus-nowcast-agrif
2024-07-21 10:09:51,064 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-07-21 10:09:51,276 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11101963.orca2.ibb'], host='localhost')
2024-07-21 10:09:52,521 INFO [watch_NEMO_agrif] running in process 714847
2024-07-21 10:09:52,522 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:09:52,524 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5565
2024-07-21 10:09:52,525 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:09:53,306 INFO [watch_NEMO_agrif] watching 21jul24nowcast-agrif job 11101963 on orcinus-nowcast-agrif
2024-07-21 10:09:53,539 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif job 11101963 is queued on orcinus-nowcast-agrif
2024-07-21 10:09:53,558 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/21jul24nowcast-green_2024-07-21T100952.448855-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-21 10:09:53,584 INFO [run_NEMO] nowcast-green NEMO run for 2024-07-21 on arbutus.cloud-nowcast started
2024-07-21 10:09:53,589 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-21 10:09:53,804 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-07-21 10:09:54,555 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-21 forecast run
2024-07-21 10:09:54,560 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-21 10:09:59,379 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-21 10:10:40,873 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-21 forecast run
2024-07-21 10:10:40,875 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-21 10:10:41,090 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 10:10:52,145 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-21 10:10:52,356 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-21 10:10:53,747 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif job 11101963 is queued on orcinus-nowcast-agrif
2024-07-21 10:11:04,170 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-21 10:11:53,954 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif job 11101963 is queued on orcinus-nowcast-agrif
2024-07-21 10:12:55,001 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-07-21 10:14:59,623 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17798560 = 2024-07-21 01:46:00 UTC, 7.4% complete
2024-07-21 10:16:04,200 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 75, 17.4% complete
2024-07-21 10:17:55,442 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7819390 = 2024-07-21 02:06:00 UTC, 8.8% complete
2024-07-21 10:19:59,611 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17798722 = 2024-07-21 03:34:00 UTC, 14.9% complete
2024-07-21 10:21:04,170 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 162, 37.5% complete
2024-07-21 10:22:55,917 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7819599 = 2024-07-21 04:25:20 UTC, 18.4% complete
2024-07-21 10:24:59,609 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17798894 = 2024-07-21 05:28:40 UTC, 22.8% complete
2024-07-21 10:26:04,148 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 237, 54.9% complete
2024-07-21 10:27:56,393 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7819811 = 2024-07-21 06:46:40 UTC, 28.3% complete
2024-07-21 10:29:59,667 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17799062 = 2024-07-21 07:20:40 UTC, 30.6% complete
2024-07-21 10:31:04,170 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 318, 73.6% complete
2024-07-21 10:32:56,837 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7820021 = 2024-07-21 09:06:40 UTC, 38.0% complete
2024-07-21 10:34:59,664 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17799235 = 2024-07-21 09:16:00 UTC, 38.6% complete
2024-07-21 10:36:04,199 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 399, 92.4% complete
2024-07-21 10:37:57,336 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7820232 = 2024-07-21 11:27:20 UTC, 47.8% complete
2024-07-21 10:39:16,441 INFO [download_results] forecast 2024-07-21 results files from arbutus.cloud-nowcast downloaded
2024-07-21 10:39:16,446 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 10:39:16,656 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:39:18,606 INFO [make_CHS_currents_file] running in process 715197
2024-07-21 10:39:18,609 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:39:18,610 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-21 10:39:18,611 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:39:58,785 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-21 for forecast
2024-07-21 10:39:58,787 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-21 10:39:58,949 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:39:58,974 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17799403 = 2024-07-21 11:08:00 UTC, 46.4% complete
2024-07-21 10:40:00,111 INFO [update_forecast_datasets] running in process 715225
2024-07-21 10:40:00,112 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:40:00,113 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2024-07-21 10:40:00,115 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:40:00,116 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/21jul24 run
2024-07-21 10:40:00,216 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/21jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-21 10:40:00,217 INFO [update_forecast_datasets] nemo 2024-07-21 forecast rolling forecast datasets updated
2024-07-21 10:40:00,219 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-21 10:40:00,424 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-21 10:40:00,429 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:40:00,434 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:40:01,529 INFO [ping_erddap] running in process 715230
2024-07-21 10:40:01,530 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:40:01,531 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-21 10:40:01,532 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:40:01,541 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-21 10:40:01,543 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-21 10:40:02,691 INFO [make_surface_current_tiles] running in process 715233
2024-07-21 10:40:02,692 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:40:02,694 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2024-07-21 10:40:02,695 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:40:02,697 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-21 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/21jul24
2024-07-21 10:40:04,412 INFO [make_plots] running in process 715232
2024-07-21 10:40:04,414 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:40:04,415 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-21 10:40:04,417 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:40:05,040 INFO [make_plots] preparing render list for 2024-07-21 NEMO forecast publish figures
2024-07-21 10:40:05,161 INFO [make_plots] added Website_thumbnail figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,162 INFO [make_plots] added Threshold_website figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,163 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,163 INFO [make_plots] added Boundary Bay figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,164 INFO [make_plots] added Campbell River figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,165 INFO [make_plots] added Cherry Point figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,166 INFO [make_plots] added Friday Harbor figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,167 INFO [make_plots] added Halfmoon Bay figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,168 INFO [make_plots] added Nanaimo figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,168 INFO [make_plots] added Neah Bay figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,169 INFO [make_plots] added New Westminster figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,170 INFO [make_plots] added Patricia Bay figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,171 INFO [make_plots] added Point Atkinson figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,172 INFO [make_plots] added Port Renfrew figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,172 INFO [make_plots] added Sand Heads figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,173 INFO [make_plots] added Sandy Cove figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,174 INFO [make_plots] added Squamish figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,175 INFO [make_plots] added Victoria figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,175 INFO [make_plots] added Woodwards Landing figure to 2024-07-21 NEMO forecast publish render list
2024-07-21 10:40:05,176 INFO [make_plots] starting to render nemo forecast publish 21jul24 figures
2024-07-21 10:41:03,514 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-21 10:41:03,517 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-21 10:41:04,052 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 10:41:04,058 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-21'], host='arbutus.cloud-nowcast')
2024-07-21 10:41:05,859 INFO [download_wwatch3_results] running in process 715555
2024-07-21 10:41:05,860 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:41:05,861 INFO [download_wwatch3_results] publishing log messages to tcp://*:5568
2024-07-21 10:41:05,862 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:41:06,043 INFO [run_ww3] running in process 4412
2024-07-21 10:41:06,044 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:41:06,045 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-07-21 10:41:06,046 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:41:06,050 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-07-21T104106.070363-0700
2024-07-21 10:41:06,051 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-07-21T104106.070363-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-07-21 10:41:06,071 INFO [run_ww3] forecast WaveWatch3 run for 2024-07-21 on arbutus.cloud-nowcast started
2024-07-21 10:41:06,074 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-21 10:41:06,485 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-21 10:41:08,527 INFO [watch_ww3] running in process 4566
2024-07-21 10:41:08,530 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:41:08,530 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-07-21 10:41:08,531 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:41:08,566 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-21 10:42:57,795 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7820443 = 2024-07-21 13:48:00 UTC, 57.5% complete
2024-07-21 10:43:05,789 INFO [download_wwatch3_results] nowcast 2024-07-21 results files from arbutus.cloud-nowcast downloaded
2024-07-21 10:43:05,792 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-21 10:44:58,974 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17799577 = 2024-07-21 13:04:00 UTC, 54.5% complete
2024-07-21 10:46:08,562 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 72, 11.1% complete
2024-07-21 10:47:58,236 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7820652 = 2024-07-21 16:07:20 UTC, 67.2% complete
2024-07-21 10:48:10,393 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 10:48:10,946 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-06-21'], host='localhost')
2024-07-21 10:48:10,952 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 10:48:10,956 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 10:48:12,682 INFO [download_results] running in process 716692
2024-07-21 10:48:12,683 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 10:48:12,684 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-21 10:48:12,685 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 10:48:12,686 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21jun18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 10:48:16,980 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 10:48:59,466 INFO [download_results] hindcast 2018-06-21 results files from optimum-hindcast downloaded
2024-07-21 10:48:59,477 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 10:48:59,890 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-06-21'], host='localhost')
2024-07-21 10:49:01,734 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 10:49:58,984 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17799737 = 2024-07-21 14:50:40 UTC, 61.9% complete
2024-07-21 10:50:48,759 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/21jul24
2024-07-21 10:51:08,576 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 147, 22.7% complete
2024-07-21 10:52:58,711 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7820864 = 2024-07-21 18:28:40 UTC, 77.0% complete
2024-07-21 10:53:05,189 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-21 10:53:05,190 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-21 10:53:45,524 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:54:58,996 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17799901 = 2024-07-21 16:40:00 UTC, 69.5% complete
2024-07-21 10:55:19,050 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/21jul24
2024-07-21 10:55:19,062 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-21 forecast into /results/nowcast-sys/figures/surface_currents/forecast/21jul24
2024-07-21 10:55:19,063 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-21 forecast completed
2024-07-21 10:55:19,379 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-21 10:55:31,809 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:55:32,163 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:56:07,043 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:56:08,586 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 228, 35.2% complete
2024-07-21 10:57:16,004 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:57:51,753 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:57:59,155 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7821076 = 2024-07-21 20:50:00 UTC, 86.8% complete
2024-07-21 10:58:27,451 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:59:02,880 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:59:36,794 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 10:59:59,003 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17800076 = 2024-07-21 18:36:40 UTC, 77.6% complete
2024-07-21 11:00:12,335 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 11:00:47,940 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 11:00:48,291 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 11:01:08,593 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete
2024-07-21 11:01:23,161 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 11:01:57,290 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-16 00:00:00Z to 2024-07-23 11:55:00Z
2024-07-21 11:02:32,553 INFO [make_plots] finished rendering nemo forecast publish 21jul24 figures
2024-07-21 11:02:32,556 INFO [make_plots] nemo publish plots for 2024-07-21 forecast completed
2024-07-21 11:02:32,563 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-21 11:02:32,774 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:02:35,543 INFO [make_feeds] running in process 717172
2024-07-21 11:02:35,546 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:02:35,547 INFO [make_feeds] publishing log messages to tcp://*:5564
2024-07-21 11:02:35,548 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:02:59,596 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7821286 = 2024-07-21 23:10:00 UTC, 96.6% complete
2024-07-21 11:03:03,287 INFO [make_feeds] ATOM feeds for 2024-07-21 forecast run completed
2024-07-21 11:03:03,291 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-21 11:04:59,009 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17800243 = 2024-07-21 20:28:00 UTC, 85.3% complete
2024-07-21 11:06:08,598 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 390, 60.2% complete
2024-07-21 11:08:00,025 INFO [watch_NEMO_agrif] 21jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7821360 = 2024-07-21 23:59:20 UTC, 100.0% complete
2024-07-21 11:09:59,013 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17800413 = 2024-07-21 22:21:20 UTC, 93.2% complete
2024-07-21 11:11:08,602 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 471, 72.7% complete
2024-07-21 11:13:00,245 INFO [watch_NEMO_agrif] NEMO AGRIF run 11101963.orca2.ibb on orcinus-nowcast-agrif completed
2024-07-21 11:13:00,249 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-07-21 11:13:00,619 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:13:02,015 INFO [download_results] running in process 717292
2024-07-21 11:13:02,016 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:13:02,018 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 11:13:02,019 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:13:02,020 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/21jul24 to /results/SalishSea/nowcast-agrif.201702
2024-07-21 11:13:25,227 INFO [download_results] nowcast-agrif 2024-07-21 results files from orcinus-nowcast-agrif downloaded
2024-07-21 11:13:25,235 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 11:13:25,404 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:13:28,670 INFO [make_plots] running in process 717300
2024-07-21 11:13:28,671 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:13:28,673 INFO [make_plots] publishing log messages to tcp://*:5564
2024-07-21 11:13:28,674 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:13:29,211 INFO [make_plots] preparing render list for 2024-07-21 NEMO nowcast-agrif research figures
2024-07-21 11:13:29,212 INFO [make_plots] added baynes_sound_surface figure to 2024-07-21 NEMO nowcast-agrif research render list
2024-07-21 11:13:29,213 INFO [make_plots] starting to render nemo nowcast-agrif research 21jul24 figures
2024-07-21 11:13:38,915 INFO [make_plots] finished rendering nemo nowcast-agrif research 21jul24 figures
2024-07-21 11:13:38,917 INFO [make_plots] nemo research plots for 2024-07-21 nowcast-agrif completed
2024-07-21 11:13:38,922 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-21 11:14:59,017 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17800560 = 2024-07-21 23:59:20 UTC, 100.0% complete
2024-07-21 11:16:08,605 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 552, 85.2% complete
2024-07-21 11:19:59,938 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-07-21 11:19:59,942 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-21 11:20:00,347 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:20:01,867 INFO [download_results] running in process 717361
2024-07-21 11:20:01,869 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:20:01,870 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-21 11:20:01,872 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:20:01,873 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/21jul24 to /results2/SalishSea/nowcast-green.202111
2024-07-21 11:21:09,259 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 630, 97.2% complete
2024-07-21 11:26:09,296 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-21 11:26:09,300 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-21 11:26:09,678 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:26:10,989 INFO [download_wwatch3_results] running in process 717456
2024-07-21 11:26:10,991 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:26:10,992 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-07-21 11:26:10,993 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:32:46,257 INFO [download_wwatch3_results] forecast 2024-07-21 results files from arbutus.cloud-nowcast downloaded
2024-07-21 11:32:46,261 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-21 11:32:46,472 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:32:47,841 INFO [update_forecast_datasets] running in process 717479
2024-07-21 11:32:47,842 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:32:47,843 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-07-21 11:32:47,845 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:32:47,845 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/21jul24 run
2024-07-21 11:32:47,847 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/21jul24 run
2024-07-21 11:32:47,858 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/21jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-21 11:32:47,859 INFO [update_forecast_datasets] wwatch3 2024-07-21 forecast rolling forecast datasets updated
2024-07-21 11:32:47,861 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-21 11:32:48,067 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-21 11:32:49,239 INFO [ping_erddap] running in process 717484
2024-07-21 11:32:49,241 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:32:49,242 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-21 11:32:49,243 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:32:49,245 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-21 11:32:49,247 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-21 11:32:49,453 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:32:53,513 INFO [make_plots] running in process 717489
2024-07-21 11:32:53,514 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:32:53,515 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-21 11:32:53,516 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:32:53,518 INFO [make_plots] preparing render list for 2024-07-21 WaveWatch3 forecast publish figures
2024-07-21 11:32:53,519 INFO [make_plots] added Halibut Bank figure to 2024-07-21 WaveWatch3 forecast publish render list
2024-07-21 11:32:53,520 INFO [make_plots] added Sentry Shoal figure to 2024-07-21 WaveWatch3 forecast publish render list
2024-07-21 11:32:53,521 INFO [make_plots] starting to render wwatch3 forecast publish 21jul24 figures
2024-07-21 11:32:53,867 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-21 11:33:01,380 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-21 11:33:07,787 INFO [make_plots] finished rendering wwatch3 forecast publish 21jul24 figures
2024-07-21 11:33:07,788 INFO [make_plots] wwatch3 publish plots for 2024-07-21 forecast completed
2024-07-21 11:33:07,792 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-21 11:57:09,458 INFO [download_results] nowcast-green 2024-07-21 results files from arbutus.cloud-nowcast downloaded
2024-07-21 11:57:09,464 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 11:57:09,671 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:57:09,676 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-07-21 11:57:09,680 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:57:09,685 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:57:09,689 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-07-21'], host='localhost')
2024-07-21 11:57:11,026 INFO [ping_erddap] running in process 717590
2024-07-21 11:57:11,027 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:57:11,028 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-21 11:57:11,028 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:57:11,033 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-07-21 11:57:11,036 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-21 11:57:12,036 INFO [make_averaged_dataset] running in process 717592
2024-07-21 11:57:12,037 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:57:12,038 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2024-07-21 11:57:12,040 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:57:12,040 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Jul-2024 chemistry
2024-07-21 11:57:12,041 INFO [make_averaged_dataset] running in process 717593
2024-07-21 11:57:12,042 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:57:12,043 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2024-07-21 11:57:12,044 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:57:12,045 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Jul-2024 physics
2024-07-21 11:57:12,069 INFO [make_averaged_dataset] running in process 717591
2024-07-21 11:57:12,070 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:57:12,071 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2024-07-21 11:57:12,072 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:57:12,073 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Jul-2024 biology
2024-07-21 11:57:13,146 INFO [make_plots] running in process 717588
2024-07-21 11:57:13,148 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 11:57:13,149 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-21 11:57:13,150 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-21 11:57:13,642 INFO [make_plots] preparing render list for 2024-07-21 NEMO nowcast-green research figures
2024-07-21 11:57:15,322 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:16,024 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:16,747 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:17,461 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:18,189 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:18,958 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:19,685 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:20,500 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:21,293 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:22,098 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:22,924 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:23,646 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:24,350 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:24,351 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:24,352 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:24,352 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-07-21 NEMO nowcast-green research render list
2024-07-21 11:57:24,353 INFO [make_plots] starting to render nemo nowcast-green research 21jul24 figures
2024-07-21 11:57:49,332 INFO [make_averaged_dataset] day-averaged dataset for 21-Jul-2024 physics created
2024-07-21 11:57:49,335 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-21 11:58:38,248 INFO [make_averaged_dataset] day-averaged dataset for 21-Jul-2024 chemistry created
2024-07-21 11:58:38,250 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-21 11:58:50,299 INFO [make_averaged_dataset] day-averaged dataset for 21-Jul-2024 biology created
2024-07-21 11:58:50,302 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-21 12:18:21,694 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 12:18:22,043 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-06-26'], host='localhost')
2024-07-21 12:18:22,051 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 12:18:22,056 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 12:18:23,673 INFO [download_results] running in process 717830
2024-07-21 12:18:23,676 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 12:18:23,677 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-21 12:18:23,678 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 12:18:23,679 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jun18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 12:18:28,027 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 12:19:15,197 INFO [download_results] hindcast 2018-06-26 results files from optimum-hindcast downloaded
2024-07-21 12:19:15,226 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 12:19:15,443 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-06-26'], host='localhost')
2024-07-21 12:19:16,918 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 12:19:17,141 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-jun', 'robot.graham'], host='localhost')
2024-07-21 12:19:18,517 INFO [archive_tarball] running in process 717876
2024-07-21 12:19:18,535 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 12:19:18,552 INFO [archive_tarball] publishing log messages to tcp://*:5565
2024-07-21 12:19:18,570 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-21 12:19:18,572 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jun18.tar from /ocean/sallen/timbucktwo/oxygen/*jun18/
2024-07-21 12:24:34,289 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jun18.index from /ocean/dlatorne/oxygen-jun18.tar
2024-07-21 12:24:34,362 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jun18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-21 12:50:02,474 INFO [archive_tarball] hindcast *jun18 results files archived to robot.graham
2024-07-21 12:50:02,477 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-21 12:58:17,909 INFO [make_plots] finished rendering nemo nowcast-green research 21jul24 figures
2024-07-21 12:58:18,086 INFO [make_plots] nemo research plots for 2024-07-21 nowcast-green completed
2024-07-21 12:58:18,144 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-21 13:43:32,292 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-21 13:43:32,638 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-07-01'], host='localhost')
2024-07-21 13:43:32,646 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 13:43:32,651 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-21 13:43:34,006 INFO [download_results] running in process 718656
2024-07-21 13:43:34,008 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 13:43:34,009 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-21 13:43:34,010 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-21 13:43:34,011 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jul18 to /ocean/sallen/timbucktwo/oxygen
2024-07-21 13:43:38,134 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-21 13:44:13,762 INFO [download_results] hindcast 2018-07-01 results files from optimum-hindcast downloaded
2024-07-21 13:44:13,773 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-21 13:44:13,982 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-07-01'], host='localhost')
2024-07-21 13:44:15,078 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-21 15:02:41,425 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240721/18/
2024-07-21 15:02:41,426 INFO [collect_weather] 2024-07-21 2.5km weather forecast 18 collection complete
2024-07-21 15:02:41,428 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-21 15:02:41,802 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-07-21 15:02:41,809 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-07-21 15:02:41,814 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-07-21 15:02:41,819 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-07-22'], host='localhost')
2024-07-21 15:02:43,105 INFO [download_weather] running in process 719027
2024-07-21 15:02:43,106 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 15:02:43,107 INFO [download_weather] publishing log messages to tcp://*:5562
2024-07-21 15:02:43,108 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-21 15:02:43,109 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240721
2024-07-21 15:02:43,110 INFO [download_weather] running in process 719029
2024-07-21 15:02:43,111 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 15:02:43,111 INFO [download_weather] publishing log messages to tcp://*:5564
2024-07-21 15:02:43,112 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-21 15:02:43,113 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240721
2024-07-21 15:02:43,114 INFO [collect_weather] running in process 719030
2024-07-21 15:02:43,115 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 15:02:43,115 INFO [collect_weather] publishing log messages to tcp://*:5565
2024-07-21 15:02:43,116 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-21 15:02:43,134 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-07-21 15:02:43,428 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240721/18/
2024-07-21 15:02:43,430 INFO [crop_gribs] 2024-07-21 18 GRIBs cropping complete
2024-07-21 15:02:43,432 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-21 15:02:44,387 INFO [crop_gribs] running in process 719031
2024-07-21 15:02:44,388 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 15:02:44,389 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-07-21 15:02:44,391 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-21 15:02:44,392 INFO [crop_gribs] cropping 2024-07-22 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-07-21 15:02:44,405 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240722/00/
2024-07-21 15:07:07,654 INFO [download_weather] 2024-07-21 1km weather forecast 00 downloads complete
2024-07-21 15:07:07,657 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-21 15:07:16,306 INFO [download_weather] 2024-07-21 1km weather forecast 12 downloads complete
2024-07-21 15:07:16,309 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-21 21:03:11,884 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240722/00/
2024-07-21 21:03:11,886 INFO [collect_weather] 2024-07-22 2.5km weather forecast 00 collection complete
2024-07-21 21:03:11,888 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-21 21:03:12,230 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-07-21 21:03:12,236 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-07-22'], host='localhost')
2024-07-21 21:03:12,652 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240722/00/
2024-07-21 21:03:12,653 INFO [crop_gribs] 2024-07-22 00 GRIBs cropping complete
2024-07-21 21:03:12,655 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-21 21:03:13,460 INFO [collect_weather] running in process 720920
2024-07-21 21:03:13,461 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 21:03:13,462 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-21 21:03:13,463 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-21 21:03:13,479 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-07-21 21:03:14,290 INFO [crop_gribs] running in process 720922
2024-07-21 21:03:14,291 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-21 21:03:14,293 INFO [crop_gribs] publishing log messages to tcp://*:5564
2024-07-21 21:03:14,294 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-21 21:03:14,295 INFO [crop_gribs] cropping 2024-07-22 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-07-21 21:03:14,309 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240722/06/
2024-07-22 03:02:09,212 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240722/06/
2024-07-22 03:02:09,214 INFO [collect_weather] 2024-07-22 2.5km weather forecast 06 collection complete
2024-07-22 03:02:09,216 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-22 03:02:09,579 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,585 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,591 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,594 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,601 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,606 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,610 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,614 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,618 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,626 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,630 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,643 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,647 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,652 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,659 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-07-22 03:02:09,666 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-07-22 03:02:09,674 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-07-22 03:02:09,689 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-07-21'], host='localhost')
2024-07-22 03:02:09,695 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-07-22 03:02:09,720 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-07-22 03:02:09,725 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-07-22 03:02:10,544 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240722/06/
2024-07-22 03:02:10,545 INFO [crop_gribs] 2024-07-22 06 GRIBs cropping complete
2024-07-22 03:02:10,547 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-22 03:02:11,313 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-07-22 03:02:12,661 INFO [collect_weather] running in process 722874
2024-07-22 03:02:12,671 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:12,672 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-22 03:02:12,674 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:12,706 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-07-22 03:02:12,769 INFO [collect_NeahBay_ssh] running in process 722873
2024-07-22 03:02:12,770 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:12,770 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-07-22 03:02:12,771 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:12,772 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-07-22
2024-07-22 03:02:13,738 INFO [collect_river_data] running in process 722858
2024-07-22 03:02:13,739 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,740 INFO [collect_river_data] publishing log messages to tcp://*:5565
2024-07-22 03:02:13,740 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,741 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-07-21
2024-07-22 03:02:13,754 INFO [collect_river_data] running in process 722856
2024-07-22 03:02:13,755 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,755 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-07-22 03:02:13,757 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,757 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-07-21
2024-07-22 03:02:13,785 INFO [collect_river_data] running in process 722862
2024-07-22 03:02:13,786 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,787 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-22 03:02:13,788 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,789 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-07-21
2024-07-22 03:02:13,811 INFO [collect_river_data] running in process 722866
2024-07-22 03:02:13,812 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,814 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-22 03:02:13,815 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,816 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-07-21
2024-07-22 03:02:13,898 INFO [collect_river_data] running in process 722861
2024-07-22 03:02:13,902 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,903 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-22 03:02:13,904 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,905 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-07-21
2024-07-22 03:02:13,907 INFO [collect_river_data] running in process 722857
2024-07-22 03:02:13,908 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,909 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-22 03:02:13,911 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,911 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-07-21
2024-07-22 03:02:13,912 INFO [collect_river_data] running in process 722868
2024-07-22 03:02:13,913 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,914 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-22 03:02:13,914 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,915 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-07-21
2024-07-22 03:02:13,923 INFO [collect_river_data] running in process 722853
2024-07-22 03:02:13,924 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,926 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-07-22 03:02:13,927 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,928 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-07-21
2024-07-22 03:02:13,930 INFO [collect_river_data] running in process 722865
2024-07-22 03:02:13,931 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,932 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-07-22 03:02:13,933 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,934 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-07-21
2024-07-22 03:02:13,938 INFO [collect_river_data] running in process 722863
2024-07-22 03:02:13,939 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,940 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-07-22 03:02:13,941 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,941 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-07-21
2024-07-22 03:02:13,946 INFO [collect_river_data] running in process 722860
2024-07-22 03:02:13,946 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,947 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-07-22 03:02:13,949 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,950 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-07-21
2024-07-22 03:02:13,953 INFO [collect_river_data] running in process 722867
2024-07-22 03:02:13,954 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,955 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-07-22 03:02:13,955 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,956 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-07-21
2024-07-22 03:02:13,961 INFO [collect_river_data] running in process 722864
2024-07-22 03:02:13,962 INFO [collect_river_data] running in process 722855
2024-07-22 03:02:13,963 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,969 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:13,969 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-07-22 03:02:13,971 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-07-22 03:02:13,971 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,972 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:13,973 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-07-21
2024-07-22 03:02:13,974 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-07-21
2024-07-22 03:02:14,055 INFO [collect_NeahBay_ssh] 2024-07-22T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-07-22 03:02:14,058 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-22 03:02:14,198 INFO [get_vfpa_hadcp] running in process 722872
2024-07-22 03:02:14,199 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:14,200 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2024-07-22 03:02:14,201 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:14,202 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-21
2024-07-22 03:02:14,342 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-07-22 03:02:14,343 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-07-21 completed
2024-07-22 03:02:14,370 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-07-21 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-07-22 03:02:14,371 INFO [collect_river_data] ECCC Fraser river data collection for 2024-07-21 completed
2024-07-22 03:02:14,521 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-07-22 03:02:14,522 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-07-21 completed
2024-07-22 03:02:14,557 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-07-22 03:02:14,558 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-07-21 completed
2024-07-22 03:02:14,578 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 03:02:14,630 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-07-22 03:02:14,631 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-07-21 completed
2024-07-22 03:02:14,634 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:14,649 INFO [crop_gribs] running in process 722875
2024-07-22 03:02:14,650 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:14,651 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-07-22 03:02:14,652 INFO [crop_gribs] publishing log messages to tcp://*:5590
2024-07-22 03:02:14,652 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-07-21 completed
2024-07-22 03:02:14,653 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:14,655 INFO [crop_gribs] cropping 2024-07-22 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-07-22 03:02:14,657 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-07-22 03:02:14,658 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-07-21 completed
2024-07-22 03:02:14,683 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240722/12/
2024-07-22 03:02:14,706 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-07-21 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-07-22 03:02:14,706 INFO [collect_river_data] ECCC Capilano river data collection for 2024-07-21 completed
2024-07-22 03:02:14,726 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-07-22 03:02:14,727 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-07-21 completed
2024-07-22 03:02:14,738 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-07-22 03:02:14,739 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-07-21 completed
2024-07-22 03:02:14,764 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-07-21 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-07-22 03:02:14,765 INFO [collect_river_data] ECCC Englishman river data collection for 2024-07-21 completed
2024-07-22 03:02:14,787 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-07-22 03:02:14,788 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-07-21 completed
2024-07-22 03:02:14,796 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-07-22 03:02:14,797 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-07-21 completed
2024-07-22 03:02:14,871 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-07-22 03:02:14,872 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-07-21 completed
2024-07-22 03:02:14,994 INFO [get_onc_ctd] running in process 722869
2024-07-22 03:02:14,995 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:14,996 INFO [get_onc_ctd] publishing log messages to tcp://*:5564
2024-07-22 03:02:14,997 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:14,998 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-07-21
2024-07-22 03:02:15,015 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:15,039 INFO [get_onc_ctd] running in process 722870
2024-07-22 03:02:15,041 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:15,041 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2024-07-22 03:02:15,042 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:15,043 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-07-21
2024-07-22 03:02:15,059 INFO [get_onc_ferry] running in process 722871
2024-07-22 03:02:15,060 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:15,061 INFO [get_onc_ferry] publishing log messages to tcp://*:5592
2024-07-22 03:02:15,062 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:15,063 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-07-21
2024-07-22 03:02:15,215 INFO [grib_to_netcdf] running in process 723102
2024-07-22 03:02:15,216 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:15,217 INFO [grib_to_netcdf] publishing log messages to tcp://*:5593
2024-07-22 03:02:15,218 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:15,219 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-22 forecast2 run
2024-07-22 03:02:15,233 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:15,422 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:15,616 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:15,819 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:16,031 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:16,312 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:16,523 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-21 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202407.nc
2024-07-22 03:02:16,524 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-07 netcdf file
2024-07-22 03:02:16,526 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:16,711 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:16,910 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-07-21
2024-07-22 03:02:16,940 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:17,140 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:17,333 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:17,354 CRITICAL [get_onc_ctd] 2024-07-21 ONC SCVIP CTD T&S file creation failed
2024-07-22 03:02:17,595 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 03:02:17,692 INFO [make_ssh_files] running in process 723299
2024-07-22 03:02:17,693 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:17,695 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2024-07-22 03:02:17,696 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:17,697 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-07-22 NOAA Neah Bay 00Z observation and forecast values
2024-07-22 03:02:17,756 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2024-07-22 03:02:17,942 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2024-07-22 03:02:19,272 INFO [ping_erddap] running in process 723354
2024-07-22 03:02:19,274 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:19,275 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-22 03:02:19,276 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:19,277 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2024-07-22 03:02:19,279 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 03:02:22,417 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d17.nc
2024-07-22 03:02:22,586 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc
2024-07-22 03:02:22,761 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc
2024-07-22 03:02:22,930 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-22 03:02:23,095 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-22 03:02:23,292 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d22.nc
2024-07-22 03:02:23,450 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d23.nc
2024-07-22 03:02:23,622 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d24.nc
2024-07-22 03:02:23,707 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d23.nc
2024-07-22 03:02:23,793 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d25.nc
2024-07-22 03:02:23,911 INFO [make_ssh_files] sea surface height boundary file for 2024-07-22 forecast2 run created
2024-07-22 03:02:23,915 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-22 03:02:24,125 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-22 03:02:24,130 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-22 03:02:25,479 INFO [make_runoff_file] running in process 723384
2024-07-22 03:02:25,480 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:25,482 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-07-22 03:02:25,483 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:25,508 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-22 03:02:25,585 INFO [make_runoff_file] 2024-07-22 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-22 03:02:25,587 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-22 03:02:26,169 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d24.nc
2024-07-22 03:02:26,175 INFO [grib_to_netcdf] 2024-07-22 NEMO-atmos forcing file for forecast2 created
2024-07-22 03:02:26,178 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-22 03:02:26,202 INFO [make_v202111_runoff_file] running in process 723382
2024-07-22 03:02:26,203 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:26,204 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564
2024-07-22 03:02:26,205 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:26,206 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-21
2024-07-22 03:02:27,146 ERROR [make_v202111_runoff_file] no 2024-07-21 discharge obs for Skagit_MountVernon: patching
2024-07-22 03:02:27,662 ERROR [make_v202111_runoff_file] no 2024-07-21 discharge obs for Snohomish_Monroe: patching
2024-07-22 03:02:27,872 ERROR [make_v202111_runoff_file] no 2024-07-21 discharge obs for Nisqually_McKenna: patching
2024-07-22 03:02:27,972 ERROR [make_v202111_runoff_file] no 2024-07-21 discharge obs for Greenwater_Greenwater: patching
2024-07-22 03:02:29,381 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-21: /results/forcing/rivers/R202108Dailies_y2024m07d21.nc
2024-07-22 03:02:29,384 INFO [make_v202111_runoff_file] 2024-07-21 runoff file creation completed
2024-07-22 03:02:29,386 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-22 03:02:29,591 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-07-22 03:02:29,595 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-07-22 03:02:29,602 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-07-22 03:02:29,608 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-07-22 03:02:30,835 INFO [upload_forcing] running in process 723426
2024-07-22 03:02:30,836 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:30,837 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-07-22 03:02:30,838 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:30,880 INFO [upload_forcing] running in process 723423
2024-07-22 03:02:30,881 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:30,882 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-07-22 03:02:30,883 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:30,885 INFO [upload_forcing] running in process 723425
2024-07-22 03:02:30,886 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:30,887 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-22 03:02:30,888 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:30,907 INFO [upload_forcing] running in process 723424
2024-07-22 03:02:30,908 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:30,909 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-22 03:02:30,910 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:32,192 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d21.nc
2024-07-22 03:02:32,252 INFO [upload_forcing] forecast2 2024-07-22 forcing files upload to optimum-hindcast completed
2024-07-22 03:02:32,257 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 03:02:32,387 INFO [upload_forcing] forecast2 2024-07-22 forcing files upload to orcinus-nowcast-agrif completed
2024-07-22 03:02:32,505 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 03:02:33,937 INFO [upload_forcing] forecast2 2024-07-22 forcing files upload to arbutus.cloud-nowcast completed
2024-07-22 03:02:33,942 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 03:02:34,153 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 03:02:34,577 WARNING [get_onc_ctd] no SEVIP salinity data
2024-07-22 03:02:34,843 INFO [get_onc_ctd] 2024-07-21 ONC SEVIP CTD T&S file created
2024-07-22 03:02:34,846 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-07-22 03:02:35,047 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-07-21
2024-07-22 03:02:35,061 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-07-22 03:02:35,335 INFO [make_forcing_links] running in process 723453
2024-07-22 03:02:35,337 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:35,338 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-22 03:02:35,339 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:36,095 WARNING [get_onc_ferry] No ONC TWDP TSG temperature,conductivity,salinity data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:36,101 WARNING [get_onc_ferry] No ONC TWDP TSG temperature data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:36,103 WARNING [get_onc_ferry] No ONC TWDP TSG conductivity data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:36,105 WARNING [get_onc_ferry] No ONC TWDP TSG salinity data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:36,106 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-07-21
2024-07-22 03:02:36,164 INFO [ping_erddap] running in process 723456
2024-07-22 03:02:36,165 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:36,166 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-22 03:02:36,167 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:36,169 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-07-22 03:02:36,171 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 03:02:36,606 INFO [make_forcing_links] forecast2 2024-07-22 forcing file links on arbutus.cloud-nowcast created
2024-07-22 03:02:36,611 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-22 03:02:36,823 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 03:02:37,010 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation,oxygen_corrected,temperature data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:37,015 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:37,017 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:37,019 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR temperature data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:37,020 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-07-21
2024-07-22 03:02:37,916 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence,chlorophyll,turbidity data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:37,922 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:37,924 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL chlorophyll data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:37,925 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL turbidity data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:37,926 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-07-21
2024-07-22 03:02:38,539 INFO [upload_forcing] forecast2 2024-07-22 forcing files upload to robot.graham completed
2024-07-22 03:02:38,544 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 03:02:39,037 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:39,042 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:39,044 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2024-07-21; substituting empty dataset
2024-07-22 03:02:39,045 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-07-21
2024-07-22 03:02:39,944 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/21jul24forecast2_2024-07-22T030239.081841-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-22 03:02:39,962 INFO [run_NEMO] forecast2 NEMO run for 2024-07-22 on arbutus.cloud-nowcast started
2024-07-22 03:02:39,967 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-22 03:02:40,175 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-22 03:02:42,108 INFO [watch_NEMO] running in process 13152
2024-07-22 03:02:42,110 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:02:42,111 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-22 03:02:42,112 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:02:42,180 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-22 03:02:56,528 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-07-21
2024-07-22 03:03:05,189 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-07-21
2024-07-22 03:03:13,779 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-07-21; substituting empty dataset
2024-07-22 03:03:13,779 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-07-21
2024-07-22 03:03:21,794 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-21; substituting empty dataset
2024-07-22 03:03:21,796 WARNING [get_onc_ferry] ONC TWDP TSG temperature data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:21,817 WARNING [get_onc_ferry] ONC TWDP TSG conductivity data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:21,879 WARNING [get_onc_ferry] ONC TWDP TSG salinity data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:21,897 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:21,914 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:21,931 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR temperature data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:21,949 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:21,966 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL chlorophyll data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:21,983 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL turbidity data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:22,001 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:22,018 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:22,120 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:03:22,138 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-22 03:04:37,524 INFO [get_onc_ferry] 2024-07-21 ONC TWDP ferry data file created
2024-07-22 03:04:37,527 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-07-22 03:04:37,723 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-07-22 03:04:39,020 INFO [ping_erddap] running in process 723478
2024-07-22 03:04:39,022 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:04:39,023 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-22 03:04:39,024 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:04:39,026 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-07-22 03:04:39,027 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 03:07:42,183 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17803162 = 2024-07-23 04:54:00 UTC, 16.3% complete
2024-07-22 03:12:42,182 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17803655 = 2024-07-23 10:22:40 UTC, 34.6% complete
2024-07-22 03:17:42,177 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17804167 = 2024-07-23 16:04:00 UTC, 53.6% complete
2024-07-22 03:22:42,174 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17804643 = 2024-07-23 21:21:20 UTC, 71.2% complete
2024-07-22 03:27:42,171 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17805150 = 2024-07-24 02:59:20 UTC, 90.0% complete
2024-07-22 03:32:42,175 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-07-22 03:32:42,179 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-22 03:32:42,535 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 03:32:42,542 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 03:32:42,547 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-21'], host='localhost')
2024-07-22 03:32:43,926 INFO [download_results] running in process 723562
2024-07-22 03:32:43,927 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:32:43,929 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-22 03:32:43,930 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:32:43,931 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/21jul24 to /results/SalishSea/forecast2.202111
2024-07-22 03:32:46,126 INFO [make_ww3_wind_file] running in process 13531
2024-07-22 03:32:46,128 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:32:46,129 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-22 03:32:46,130 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:32:46,132 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-22 forecast2 run
2024-07-22 03:32:46,179 INFO [make_ww3_current_file] running in process 13533
2024-07-22 03:32:46,181 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:32:46,182 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-22 03:32:46,183 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:32:46,185 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-22 forecast2 run
2024-07-22 03:32:55,445 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-22 forecast2 run
2024-07-22 03:32:55,448 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-22 03:34:57,770 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-22 forecast2 run
2024-07-22 03:34:57,774 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-22 03:34:57,986 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 03:35:04,272 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-22 03:35:04,500 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-22 03:35:14,939 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-22 03:40:14,440 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 75, 13.9% complete
2024-07-22 03:45:14,367 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 156, 28.9% complete
2024-07-22 03:50:14,422 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 243, 45.0% complete
2024-07-22 03:55:14,458 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 324, 60.0% complete
2024-07-22 03:57:39,449 INFO [download_results] forecast2 2024-07-21 results files from arbutus.cloud-nowcast downloaded
2024-07-22 03:57:39,453 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 03:57:39,673 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-07-21'], host='localhost')
2024-07-22 03:57:41,953 INFO [make_CHS_currents_file] running in process 723622
2024-07-22 03:57:41,954 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:57:41,955 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-22 03:57:41,956 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:58:17,079 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-21 for forecast2
2024-07-22 03:58:17,082 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-22 03:58:17,277 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-07-21'], host='localhost')
2024-07-22 03:58:18,386 INFO [update_forecast_datasets] running in process 723646
2024-07-22 03:58:18,387 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 03:58:18,389 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2024-07-22 03:58:18,390 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-22 03:58:18,391 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/21jul24 run
2024-07-22 04:00:14,282 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 405, 75.0% complete
2024-07-22 04:02:28,520 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/21jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-22 04:02:28,521 INFO [update_forecast_datasets] nemo 2024-07-21 forecast2 rolling forecast datasets updated
2024-07-22 04:02:28,524 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-22 04:02:28,736 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-22 04:02:28,742 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-07-21'], host='localhost')
2024-07-22 04:02:28,746 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-07-21'], host='localhost')
2024-07-22 04:02:30,065 INFO [ping_erddap] running in process 723723
2024-07-22 04:02:30,066 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:02:30,067 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-22 04:02:30,068 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:02:30,077 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-22 04:02:30,080 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 04:02:31,242 INFO [make_surface_current_tiles] running in process 723726
2024-07-22 04:02:31,243 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:02:31,244 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2024-07-22 04:02:31,246 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:02:31,247 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-21 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/21jul24
2024-07-22 04:02:33,093 INFO [make_plots] running in process 723725
2024-07-22 04:02:33,101 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:02:33,102 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-22 04:02:33,103 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:02:33,737 INFO [make_plots] preparing render list for 2024-07-21 NEMO forecast2 publish figures
2024-07-22 04:02:33,855 INFO [make_plots] added Website_thumbnail figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,856 INFO [make_plots] added Threshold_website figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,856 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,857 INFO [make_plots] added Boundary Bay figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,858 INFO [make_plots] added Campbell River figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,859 INFO [make_plots] added Cherry Point figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,860 INFO [make_plots] added Friday Harbor figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,861 INFO [make_plots] added Halfmoon Bay figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,861 INFO [make_plots] added Nanaimo figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,862 INFO [make_plots] added Neah Bay figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,863 INFO [make_plots] added New Westminster figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,864 INFO [make_plots] added Patricia Bay figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,864 INFO [make_plots] added Point Atkinson figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,865 INFO [make_plots] added Port Renfrew figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,866 INFO [make_plots] added Sand Heads figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,867 INFO [make_plots] added Sandy Cove figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,867 INFO [make_plots] added Squamish figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,868 INFO [make_plots] added Victoria figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,869 INFO [make_plots] added Woodwards Landing figure to 2024-07-21 NEMO forecast2 publish render list
2024-07-22 04:02:33,870 INFO [make_plots] starting to render nemo forecast2 publish 21jul24 figures
2024-07-22 04:05:14,269 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 480, 88.9% complete
2024-07-22 04:10:14,262 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-22 04:13:56,813 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/21jul24
2024-07-22 04:15:14,258 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-22 04:15:14,261 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-22 04:15:14,624 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 04:15:15,897 INFO [download_wwatch3_results] running in process 725676
2024-07-22 04:15:15,898 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:15:15,899 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-07-22 04:15:15,900 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:16:03,144 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-22 04:16:03,145 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-22 04:16:40,943 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:17:34,122 INFO [download_wwatch3_results] forecast2 2024-07-22 results files from arbutus.cloud-nowcast downloaded
2024-07-22 04:17:34,126 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-22 04:17:34,335 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 04:17:35,533 INFO [update_forecast_datasets] running in process 725705
2024-07-22 04:17:35,535 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:17:35,536 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566
2024-07-22 04:17:35,537 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:17:35,538 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/22jul24 run
2024-07-22 04:17:35,539 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/22jul24 run
2024-07-22 04:17:59,102 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/22jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-22 04:17:59,103 INFO [update_forecast_datasets] wwatch3 2024-07-22 forecast2 rolling forecast datasets updated
2024-07-22 04:17:59,106 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-22 04:17:59,291 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-22 04:18:00,503 INFO [ping_erddap] running in process 725719
2024-07-22 04:18:00,504 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:18:00,505 INFO [ping_erddap] publishing log messages to tcp://*:5566
2024-07-22 04:18:00,506 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:18:00,507 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-22 04:18:00,509 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 04:18:00,734 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 04:18:04,391 INFO [make_plots] running in process 725724
2024-07-22 04:18:04,393 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:18:04,394 INFO [make_plots] publishing log messages to tcp://*:5566
2024-07-22 04:18:04,395 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:18:04,396 INFO [make_plots] preparing render list for 2024-07-22 WaveWatch3 forecast2 publish figures
2024-07-22 04:18:04,397 INFO [make_plots] added Halibut Bank figure to 2024-07-22 WaveWatch3 forecast2 publish render list
2024-07-22 04:18:04,398 INFO [make_plots] added Sentry Shoal figure to 2024-07-22 WaveWatch3 forecast2 publish render list
2024-07-22 04:18:04,399 INFO [make_plots] starting to render wwatch3 forecast2 publish 22jul24 figures
2024-07-22 04:18:04,720 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-22 04:18:11,026 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-22 04:18:16,504 INFO [make_plots] finished rendering wwatch3 forecast2 publish 22jul24 figures
2024-07-22 04:18:16,505 INFO [make_plots] wwatch3 publish plots for 2024-07-22 forecast2 completed
2024-07-22 04:18:16,509 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-22 04:18:18,092 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/21jul24
2024-07-22 04:18:18,103 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-21 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/21jul24
2024-07-22 04:18:18,105 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-21 forecast2 completed
2024-07-22 04:18:18,408 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-22 04:18:36,052 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:18:36,404 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:19:11,872 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:20:21,698 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:20:57,490 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:21:33,246 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:22:09,305 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:22:43,501 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:23:19,519 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:23:55,339 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:23:55,696 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:24:31,160 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:25:05,434 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-17 00:00:00Z to 2024-07-24 05:55:00Z
2024-07-22 04:25:41,392 INFO [make_plots] finished rendering nemo forecast2 publish 21jul24 figures
2024-07-22 04:25:41,394 INFO [make_plots] nemo publish plots for 2024-07-21 forecast2 completed
2024-07-22 04:25:41,403 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-22 04:25:41,721 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-07-21'], host='localhost')
2024-07-22 04:25:44,457 INFO [make_feeds] running in process 725793
2024-07-22 04:25:44,458 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:25:44,459 INFO [make_feeds] publishing log messages to tcp://*:5564
2024-07-22 04:25:44,460 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:26:12,372 INFO [make_feeds] ATOM feeds for 2024-07-21 forecast2 run completed
2024-07-22 04:26:12,375 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-22 04:26:12,694 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-07-22 04:26:13,764 INFO [clear_checklist] running in process 725818
2024-07-22 04:26:13,765 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:26:13,766 INFO [clear_checklist] publishing log messages to tcp://*:5564
2024-07-22 04:26:13,767 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:26:13,768 INFO [clear_checklist] requesting that manager clear system state checklist
2024-07-22 04:26:13,770 INFO [manager] writing checklist to log file
2024-07-22 04:26:13,822 INFO [manager] checklist cleared
2024-07-22 04:26:13,824 INFO [clear_checklist] nowcast system checklist cleared
2024-07-22 04:26:13,828 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-07-22 04:26:14,857 INFO [rotate_logs] running in process 725823
2024-07-22 04:26:14,858 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 04:26:14,859 INFO [rotate_logs] publishing log messages to tcp://*:5564
2024-07-22 04:26:14,860 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-07-22 04:26:14,861 INFO [rotate_logs] rotating log files