2024-07-25 04:26:29,832 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-07-25 04:26:29,840 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-07-25 04:26:29,846 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-07-25 04:26:29,848 INFO [rotate_logs] log files rotated
2024-07-25 04:26:29,853 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-07-25 05:18:10,687 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 05:18:10,845 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-01'], host='localhost')
2024-07-25 05:18:10,853 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 05:18:10,859 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 05:18:12,114 INFO [download_results] running in process 768305
2024-07-25 05:18:12,115 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 05:18:12,116 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-25 05:18:12,118 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 05:18:12,119 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01feb19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 05:18:16,371 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 05:18:51,906 INFO [download_results] hindcast 2019-02-01 results files from optimum-hindcast downloaded
2024-07-25 05:18:51,916 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 05:18:51,924 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-01'], host='localhost')
2024-07-25 05:18:53,321 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 06:43:20,742 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 06:43:20,898 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-06'], host='localhost')
2024-07-25 06:43:20,906 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 06:43:20,911 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 06:43:22,160 INFO [download_results] running in process 770597
2024-07-25 06:43:22,162 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 06:43:22,163 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-25 06:43:22,164 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 06:43:22,165 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06feb19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 06:43:26,435 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 06:44:02,241 INFO [download_results] hindcast 2019-02-06 results files from optimum-hindcast downloaded
2024-07-25 06:44:02,250 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 06:44:02,256 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-06'], host='localhost')
2024-07-25 06:44:03,785 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 08:08:30,847 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 08:08:31,008 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-11'], host='localhost')
2024-07-25 08:08:31,015 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 08:08:31,021 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 08:08:32,238 INFO [download_results] running in process 771505
2024-07-25 08:08:32,239 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 08:08:32,241 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-25 08:08:32,242 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 08:08:32,243 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11feb19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 08:08:36,499 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 08:09:12,051 INFO [download_results] hindcast 2019-02-11 results files from optimum-hindcast downloaded
2024-07-25 08:09:12,061 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 08:09:12,070 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-11'], host='localhost')
2024-07-25 08:09:13,406 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 09:27:27,692 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240725/12/
2024-07-25 09:27:27,693 INFO [collect_weather] 2024-07-25 2.5km weather forecast 12 collection complete
2024-07-25 09:27:27,697 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-25 09:27:27,881 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 09:27:27,888 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 09:27:27,893 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 09:27:27,898 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 09:27:27,904 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-07-25 09:27:27,908 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-07-25 09:27:27,912 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-07-25 09:27:27,916 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-07-25 09:27:27,921 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-07-25 09:27:28,873 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240725/12/
2024-07-25 09:27:28,873 INFO [crop_gribs] 2024-07-25 12 GRIBs cropping complete
2024-07-25 09:27:28,875 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-25 09:27:29,026 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-07-25 09:27:29,655 INFO [collect_weather] running in process 772200
2024-07-25 09:27:29,656 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:29,657 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-25 09:27:29,658 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:29,671 INFO [collect_NeahBay_ssh] running in process 772198
2024-07-25 09:27:29,672 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:29,673 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-07-25 09:27:29,674 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:29,675 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-07-25
2024-07-25 09:27:29,680 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-07-25 09:27:30,183 INFO [collect_river_data] running in process 772196
2024-07-25 09:27:30,183 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:30,184 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-07-25 09:27:30,185 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:30,186 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-07-24
2024-07-25 09:27:30,191 INFO [collect_river_data] running in process 772195
2024-07-25 09:27:30,192 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:30,194 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-25 09:27:30,195 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:30,195 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-07-24
2024-07-25 09:27:30,214 INFO [download_live_ocean] running in process 772199
2024-07-25 09:27:30,215 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:30,216 INFO [download_live_ocean] publishing log messages to tcp://*:5568
2024-07-25 09:27:30,217 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:30,218 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-07-25
2024-07-25 09:27:30,226 INFO [collect_river_data] running in process 772194
2024-07-25 09:27:30,227 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:30,228 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-25 09:27:30,230 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:30,231 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-07-24
2024-07-25 09:27:30,328 INFO [collect_river_data] running in process 772192
2024-07-25 09:27:30,329 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:30,330 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-25 09:27:30,331 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:30,332 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-07-24
2024-07-25 09:27:30,365 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-07-24&endDT=2024-07-24 "HTTP/1.1 200 OK"
2024-07-25 09:27:30,372 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-07-24&endDT=2024-07-24 "HTTP/1.1 200 OK"
2024-07-25 09:27:30,381 INFO [make_turbidity_file] running in process 772197
2024-07-25 09:27:30,383 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:30,392 INFO [make_turbidity_file] publishing log messages to tcp://*:5571
2024-07-25 09:27:30,401 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:30,402 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-07-25
2024-07-25 09:27:30,408 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 09:27:30,410 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-07-25 09:27:30,412 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-07-25 09:27:30,413 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-07-24 completed
2024-07-25 09:27:30,414 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-07-24 completed
2024-07-25 09:27:30,417 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 09:27:30,541 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-07-24&endDT=2024-07-24 "HTTP/1.1 200 OK"
2024-07-25 09:27:30,546 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-07-25 09:27:30,547 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-07-24 completed
2024-07-25 09:27:30,549 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 09:27:30,648 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-07-24&endDT=2024-07-24 "HTTP/1.1 200 OK"
2024-07-25 09:27:30,652 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-07-25 09:27:30,653 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-07-24 completed
2024-07-25 09:27:30,655 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 09:27:30,745 INFO [crop_gribs] running in process 772201
2024-07-25 09:27:30,746 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:30,747 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-07-25 09:27:30,748 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:30,749 INFO [crop_gribs] cropping 2024-07-25 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-07-25 09:27:30,761 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240725/18/
2024-07-25 09:27:31,155 INFO [grib_to_netcdf] running in process 772289
2024-07-25 09:27:31,156 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:31,157 INFO [grib_to_netcdf] publishing log messages to tcp://*:5567
2024-07-25 09:27:31,158 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:31,160 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-25 nowcast and forecast runs
2024-07-25 09:27:31,315 INFO [collect_NeahBay_ssh] 2024-07-25T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-07-25 09:27:31,317 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-25 09:27:31,326 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 09:27:31,390 INFO [download_live_ocean] downloaded 10103856 bytes from https://liveocean.apl.uw.edu/output/f2024.07.25/ubc.nc to /results/forcing/LiveOcean/downloaded/20240725/low_passed_UBC.nc
2024-07-25 09:27:31,392 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-07-25 09:27:32,421 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240725/low_passed_UBC.nc
2024-07-25 09:27:32,422 INFO [download_live_ocean] 2024-07-25 Live Ocean file for Salish Sea western boundary downloaded
2024-07-25 09:27:32,424 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-07-25 09:27:32,576 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-07-25'], host='localhost')
2024-07-25 09:27:33,526 INFO [make_turbidity_file] 2024-07-25 Fraser River turbidity file creation complete
2024-07-25 09:27:33,529 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-07-25 09:27:34,131 INFO [make_ssh_files] running in process 772373
2024-07-25 09:27:34,132 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:34,133 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2024-07-25 09:27:34,135 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:34,136 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-07-25 NOAA Neah Bay 06Z observation and forecast values
2024-07-25 09:27:34,837 INFO [make_live_ocean_files] running in process 772388
2024-07-25 09:27:34,838 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:34,840 INFO [make_live_ocean_files] publishing log messages to tcp://*:5568
2024-07-25 09:27:34,841 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:34,846 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-07-25 Live Ocean run
2024-07-25 09:27:38,399 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-25 09:27:38,576 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d22.nc
2024-07-25 09:27:38,750 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d23.nc
2024-07-25 09:27:38,929 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d24.nc
2024-07-25 09:27:39,126 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d25.nc
2024-07-25 09:27:39,306 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d26.nc
2024-07-25 09:27:39,477 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d27.nc
2024-07-25 09:27:39,658 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d28.nc
2024-07-25 09:27:39,768 INFO [make_ssh_files] sea surface height boundary file for 2024-07-25 nowcast run created
2024-07-25 09:27:39,771 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-25 09:27:39,783 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-25 09:27:39,787 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-25 09:27:40,357 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m07d25.nc
2024-07-25 09:27:41,078 INFO [make_runoff_file] running in process 772444
2024-07-25 09:27:41,079 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:41,080 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-07-25 09:27:41,081 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:41,106 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-25 09:27:41,198 INFO [make_runoff_file] 2024-07-25 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-25 09:27:41,200 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-25 09:27:41,676 INFO [make_v202111_runoff_file] running in process 772442
2024-07-25 09:27:41,678 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:27:41,679 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564
2024-07-25 09:27:41,680 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:27:41,681 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-24
2024-07-25 09:27:44,261 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-24: /results/forcing/rivers/R202108Dailies_y2024m07d24.nc
2024-07-25 09:27:44,262 INFO [make_v202111_runoff_file] 2024-07-24 runoff file creation completed
2024-07-25 09:27:44,265 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-25 09:27:47,281 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d26.nc
2024-07-25 09:27:50,972 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d27.nc
2024-07-25 09:27:50,986 INFO [grib_to_netcdf] 2024-07-25 NEMO-atmos forcing file for nowcast+ created
2024-07-25 09:27:50,989 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-25 09:28:15,436 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d25.nc
2024-07-25 09:28:15,437 INFO [make_live_ocean_files] 2024-07-25 Live Ocean western boundary conditions files created
2024-07-25 09:28:15,439 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-07-25 09:28:15,451 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-07-25 09:28:15,456 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-07-25 09:28:15,462 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-07-25 09:28:15,469 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-07-25 09:28:15,477 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-07-25 09:28:16,677 INFO [ping_erddap] running in process 772483
2024-07-25 09:28:16,678 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:28:16,680 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-25 09:28:16,681 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:28:16,683 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-07-25 09:28:16,687 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 09:28:16,912 INFO [upload_forcing] running in process 772487
2024-07-25 09:28:16,913 INFO [upload_forcing] running in process 772485
2024-07-25 09:28:16,915 INFO [upload_forcing] running in process 772488
2024-07-25 09:28:16,916 INFO [upload_forcing] running in process 772486
2024-07-25 09:28:16,918 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:28:16,920 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:28:16,922 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:28:16,924 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:28:16,941 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-25 09:28:16,942 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-25 09:28:16,943 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-25 09:28:16,944 INFO [upload_forcing] publishing log messages to tcp://*:5570
2024-07-25 09:28:16,945 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:28:16,945 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:28:16,946 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:28:16,947 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:28:18,969 INFO [upload_forcing] nowcast+ 2024-07-25 forcing files upload to optimum-hindcast completed
2024-07-25 09:28:18,973 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 09:28:19,133 INFO [upload_forcing] nowcast+ 2024-07-25 forcing files upload to orcinus-nowcast-agrif completed
2024-07-25 09:28:19,137 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 09:28:26,959 INFO [upload_forcing] nowcast+ 2024-07-25 forcing files upload to robot.graham completed
2024-07-25 09:28:26,964 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 09:28:28,957 INFO [upload_forcing] nowcast+ 2024-07-25 forcing files upload to arbutus.cloud-nowcast completed
2024-07-25 09:28:28,962 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 09:28:28,976 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 09:28:30,681 INFO [make_forcing_links] running in process 772522
2024-07-25 09:28:30,683 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:28:30,684 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-25 09:28:30,685 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:28:31,833 INFO [make_forcing_links] nowcast+ 2024-07-25 forcing file links on arbutus.cloud-nowcast created
2024-07-25 09:28:31,838 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-25 09:28:31,853 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 09:28:33,982 INFO [run_NEMO] running in process 747
2024-07-25 09:28:33,983 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:28:33,985 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-25 09:28:33,994 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:28:34,917 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/25jul24nowcast_2024-07-25T092834.038431-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-25 09:28:34,935 INFO [run_NEMO] nowcast NEMO run for 2024-07-25 on arbutus.cloud-nowcast started
2024-07-25 09:28:34,940 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-25 09:28:34,954 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-25 09:28:36,940 INFO [watch_NEMO] running in process 1008
2024-07-25 09:28:36,942 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:28:36,943 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-25 09:28:36,945 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:28:37,010 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-25 09:33:37,011 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17807490 = 2024-07-25 04:59:20 UTC, 20.8% complete
2024-07-25 09:38:37,011 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17807954 = 2024-07-25 10:08:40 UTC, 42.3% complete
2024-07-25 09:38:40,698 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 09:38:40,861 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-16'], host='localhost')
2024-07-25 09:38:40,870 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 09:38:40,876 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 09:38:42,639 INFO [download_results] running in process 772568
2024-07-25 09:38:42,640 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:38:42,642 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-25 09:38:42,643 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:38:42,645 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16feb19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 09:38:46,984 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 09:39:25,760 INFO [download_results] hindcast 2019-02-16 results files from optimum-hindcast downloaded
2024-07-25 09:39:25,771 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 09:39:25,785 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-16'], host='localhost')
2024-07-25 09:39:27,503 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 09:43:37,016 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17808428 = 2024-07-25 15:24:40 UTC, 64.2% complete
2024-07-25 09:48:37,019 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17808897 = 2024-07-25 20:37:20 UTC, 86.0% complete
2024-07-25 09:53:37,024 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-07-25 09:53:37,028 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-25 09:53:37,207 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 09:53:37,214 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 09:53:38,677 INFO [download_results] running in process 772680
2024-07-25 09:53:38,679 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:53:38,680 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-25 09:53:38,681 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:53:38,682 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/25jul24 to /results/SalishSea/nowcast-blue.202111
2024-07-25 09:53:38,684 INFO [make_forcing_links] running in process 772679
2024-07-25 09:53:38,685 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:53:38,686 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-25 09:53:38,687 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:53:39,591 INFO [make_forcing_links] ssh 2024-07-25 forcing file links on arbutus.cloud-nowcast created
2024-07-25 09:53:39,595 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-25 09:53:39,611 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 09:53:41,817 INFO [run_NEMO] running in process 1575
2024-07-25 09:53:41,818 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:53:41,820 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-25 09:53:41,821 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:53:42,907 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/25jul24forecast_2024-07-25T095341.983349-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-25 09:53:42,927 INFO [run_NEMO] forecast NEMO run for 2024-07-25 on arbutus.cloud-nowcast started
2024-07-25 09:53:42,932 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-25 09:53:42,946 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-25 09:53:45,524 INFO [watch_NEMO] running in process 1834
2024-07-25 09:53:45,525 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 09:53:45,527 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-25 09:53:45,529 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-25 09:53:45,756 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-25 09:58:46,410 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17809618 = 2024-07-26 04:38:00 UTC, 12.9% complete
2024-07-25 10:03:46,420 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17810091 = 2024-07-26 09:53:20 UTC, 27.5% complete
2024-07-25 10:08:46,397 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17810569 = 2024-07-26 15:12:00 UTC, 42.2% complete
2024-07-25 10:13:46,376 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17811048 = 2024-07-26 20:31:20 UTC, 57.0% complete
2024-07-25 10:13:54,213 INFO [download_results] nowcast 2024-07-25 results files from arbutus.cloud-nowcast downloaded
2024-07-25 10:13:54,218 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 10:13:54,232 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:13:54,239 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-07-24'], host='localhost')
2024-07-25 10:13:54,243 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:13:56,375 INFO [make_CHS_currents_file] running in process 772742
2024-07-25 10:13:56,377 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:13:56,377 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-25 10:13:56,378 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:13:57,740 INFO [make_plots] running in process 772739
2024-07-25 10:13:57,742 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:13:57,743 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-25 10:13:57,744 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:13:57,745 INFO [make_plots] running in process 772741
2024-07-25 10:13:57,746 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:13:57,748 INFO [make_plots] publishing log messages to tcp://*:5568
2024-07-25 10:13:57,754 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:13:59,114 INFO [make_plots] preparing render list for 2024-07-25 NEMO nowcast-blue research figures
2024-07-25 10:13:59,117 INFO [make_plots] preparing render list for 2024-07-24 NEMO nowcast-blue comparison figures
2024-07-25 10:14:02,064 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-blue research render list
2024-07-25 10:14:02,713 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-blue research render list
2024-07-25 10:14:02,714 INFO [make_plots] added Currents_sections_and_surface figure to 2024-07-25 NEMO nowcast-blue research render list
2024-07-25 10:14:02,715 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-07-25 NEMO nowcast-blue research render list
2024-07-25 10:14:02,716 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-07-25 NEMO nowcast-blue research render list
2024-07-25 10:14:02,717 INFO [make_plots] starting to render nemo nowcast research 25jul24 figures
2024-07-25 10:14:25,457 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-25 for nowcast
2024-07-25 10:14:25,460 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-25 10:14:58,891 INFO [make_plots] added SH_wind figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,892 INFO [make_plots] added Compare_VENUS_East figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,895 INFO [make_plots] added Compare_VENUS_Central figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,896 INFO [make_plots] added Central_ADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,896 INFO [make_plots] added Central_depavADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,897 INFO [make_plots] added Central_timeavADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,898 INFO [make_plots] added East_ADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,899 INFO [make_plots] added East_depavADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,899 INFO [make_plots] added East_timeavADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,900 INFO [make_plots] added ddl_ADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,901 INFO [make_plots] added ddl_depavADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,901 INFO [make_plots] added ddl_timeavADCP figure to 2024-07-24 NEMO nowcast-blue comparison render list
2024-07-25 10:14:58,902 INFO [make_plots] starting to render nemo nowcast comparison 24jul24 figures
2024-07-25 10:15:11,744 INFO [make_plots] finished rendering nemo nowcast research 25jul24 figures
2024-07-25 10:15:11,747 INFO [make_plots] nemo research plots for 2024-07-25 nowcast completed
2024-07-25 10:15:11,762 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-25 10:16:16,524 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available
2024-07-25 10:16:17,018 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-25 10:16:17,501 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-25 10:16:17,987 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-25 10:16:18,475 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-25 10:16:18,961 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-25 10:16:19,455 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-25 10:16:19,945 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-25 10:16:20,430 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-25 10:16:20,917 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-25 10:16:20,918 INFO [make_plots] finished rendering nemo nowcast comparison 24jul24 figures
2024-07-25 10:16:21,030 INFO [make_plots] nemo comparison plots for 2024-07-24 nowcast completed
2024-07-25 10:16:21,033 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-25 10:18:45,725 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17811514 = 2024-07-27 01:42:00 UTC, 71.4% complete
2024-07-25 10:23:45,726 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17811989 = 2024-07-27 06:58:40 UTC, 86.1% complete
2024-07-25 10:28:45,728 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17812440 = 2024-07-27 11:59:20 UTC, 100.0% complete
2024-07-25 10:33:45,731 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-07-25 10:33:45,735 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-25 10:33:45,912 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 10:33:45,918 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 10:33:45,922 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:33:45,929 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:33:45,934 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:33:45,939 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:33:45,943 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:33:47,214 INFO [upload_forcing] running in process 772906
2024-07-25 10:33:47,215 INFO [download_results] running in process 772907
2024-07-25 10:33:47,216 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:47,217 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-07-25 10:33:47,218 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:47,219 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:47,220 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-25 10:33:47,221 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:47,222 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/25jul24 to /results/SalishSea/forecast.202111
2024-07-25 10:33:47,231 INFO [upload_forcing] running in process 772905
2024-07-25 10:33:47,233 INFO [upload_forcing] running in process 772902
2024-07-25 10:33:47,234 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:47,235 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-25 10:33:47,236 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:47,237 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:47,239 INFO [upload_forcing] running in process 772904
2024-07-25 10:33:47,240 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-25 10:33:47,241 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:47,242 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:47,244 INFO [upload_forcing] publishing log messages to tcp://*:5570
2024-07-25 10:33:47,245 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:47,894 INFO [upload_forcing] turbidity 2024-07-25 forcing files upload to optimum-hindcast completed
2024-07-25 10:33:47,898 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 10:33:47,920 INFO [upload_forcing] turbidity 2024-07-25 forcing files upload to orcinus-nowcast-agrif completed
2024-07-25 10:33:47,923 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 10:33:47,938 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:33:48,234 INFO [upload_forcing] turbidity 2024-07-25 forcing files upload to arbutus.cloud-nowcast completed
2024-07-25 10:33:48,238 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 10:33:48,255 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:33:48,970 INFO [make_ww3_current_file] running in process 2470
2024-07-25 10:33:48,971 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:48,972 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-25 10:33:49,006 INFO [make_ww3_wind_file] running in process 2471
2024-07-25 10:33:49,007 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:49,008 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-25 10:33:49,039 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:49,040 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:49,041 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-25 forecast run
2024-07-25 10:33:49,042 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-25 forecast run
2024-07-25 10:33:49,244 INFO [make_forcing_links] running in process 772938
2024-07-25 10:33:49,245 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:49,246 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-25 10:33:49,247 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:49,569 INFO [make_forcing_links] running in process 772940
2024-07-25 10:33:49,570 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:49,572 INFO [make_forcing_links] publishing log messages to tcp://*:5569
2024-07-25 10:33:49,573 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:50,084 INFO [make_forcing_links] nowcast-agrif 2024-07-25 forcing file links on orcinus-nowcast-agrif created
2024-07-25 10:33:50,088 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-25 10:33:50,108 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 10:33:51,191 INFO [upload_forcing] turbidity 2024-07-25 forcing files upload to robot.graham completed
2024-07-25 10:33:51,195 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 10:33:51,475 INFO [run_NEMO_agrif] running in process 772954
2024-07-25 10:33:51,477 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:51,478 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2024-07-25 10:33:51,479 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:53,173 INFO [make_forcing_links] nowcast-green 2024-07-25 forcing file links on arbutus.cloud-nowcast created
2024-07-25 10:33:53,177 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-25 10:33:53,195 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 10:33:57,482 INFO [run_NEMO] running in process 3241
2024-07-25 10:33:57,484 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:33:57,485 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-25 10:33:57,486 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:33:58,675 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/25jul24nowcast-green_2024-07-25T103357.469514-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-25 10:33:58,700 INFO [run_NEMO] nowcast-green NEMO run for 2024-07-25 on arbutus.cloud-nowcast started
2024-07-25 10:33:58,705 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-25 10:33:58,723 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-07-25 10:34:00,118 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-25 forecast run
2024-07-25 10:34:00,122 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-25 10:34:01,435 INFO [watch_NEMO] running in process 3503
2024-07-25 10:34:01,436 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:34:01,437 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-25 10:34:01,439 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:34:01,669 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-25 10:34:06,874 INFO [run_NEMO_agrif] job id for 25jul24nowcast-agrif: 11101991.orca2.ibb
2024-07-25 10:34:06,875 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-07-25 queued on orcinus-nowcast-agrif
2024-07-25 10:34:06,879 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-07-25 10:34:06,899 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11101991.orca2.ibb'], host='localhost')
2024-07-25 10:34:08,301 INFO [watch_NEMO_agrif] running in process 772973
2024-07-25 10:34:08,302 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:34:08,304 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2024-07-25 10:34:08,305 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:34:09,096 INFO [watch_NEMO_agrif] watching 25jul24nowcast-agrif job 11101991 on orcinus-nowcast-agrif
2024-07-25 10:34:10,215 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-07-25 10:34:47,117 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-25 forecast run
2024-07-25 10:34:47,122 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-25 10:34:47,142 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 10:34:49,979 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-25 10:34:50,001 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-25 10:34:52,683 INFO [watch_ww3] running in process 3686
2024-07-25 10:34:52,683 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 10:34:52,685 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-07-25 10:34:52,687 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-25 10:34:52,932 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-25 10:39:01,709 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17807193 = 2024-07-25 01:41:20 UTC, 7.0% complete
2024-07-25 10:39:10,697 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7828004 = 2024-07-25 01:48:40 UTC, 7.5% complete
2024-07-25 10:39:52,918 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 75, 17.4% complete
2024-07-25 10:44:01,674 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17807365 = 2024-07-25 03:36:00 UTC, 15.0% complete
2024-07-25 10:44:11,187 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7828209 = 2024-07-25 04:05:20 UTC, 17.0% complete
2024-07-25 10:44:52,927 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 165, 38.2% complete
2024-07-25 10:49:01,697 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17807537 = 2024-07-25 05:30:40 UTC, 23.0% complete
2024-07-25 10:49:11,661 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7828418 = 2024-07-25 06:24:40 UTC, 26.7% complete
2024-07-25 10:49:52,964 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 261, 60.4% complete
2024-07-25 10:54:01,680 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17807709 = 2024-07-25 07:25:20 UTC, 30.9% complete
2024-07-25 10:54:12,116 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7828629 = 2024-07-25 08:45:20 UTC, 36.5% complete
2024-07-25 10:54:52,958 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 345, 79.9% complete
2024-07-25 10:59:01,686 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17807880 = 2024-07-25 09:19:20 UTC, 38.9% complete
2024-07-25 10:59:12,600 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7828836 = 2024-07-25 11:03:20 UTC, 46.1% complete
2024-07-25 10:59:52,940 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-25 11:03:14,882 INFO [download_results] forecast 2024-07-25 results files from arbutus.cloud-nowcast downloaded
2024-07-25 11:03:14,886 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 11:03:14,906 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:03:16,755 INFO [make_CHS_currents_file] running in process 773074
2024-07-25 11:03:16,756 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:03:16,757 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-07-25 11:03:16,758 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:03:51,191 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 11:03:51,437 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-21'], host='localhost')
2024-07-25 11:03:51,444 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 11:03:51,450 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 11:03:52,557 INFO [download_results] running in process 773102
2024-07-25 11:03:52,558 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:03:52,559 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-25 11:03:52,561 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:03:52,562 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21feb19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 11:03:56,906 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 11:03:58,422 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-25 for forecast
2024-07-25 11:03:58,425 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-25 11:03:58,450 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:04:00,217 INFO [update_forecast_datasets] running in process 773141
2024-07-25 11:04:00,219 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:04:00,220 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-07-25 11:04:00,221 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:04:00,221 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/25jul24 run
2024-07-25 11:04:00,553 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/25jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-25 11:04:00,558 INFO [update_forecast_datasets] nemo 2024-07-25 forecast rolling forecast datasets updated
2024-07-25 11:04:00,571 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-25 11:04:00,576 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-25 11:04:00,582 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:04:00,588 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:04:01,635 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17808051 = 2024-07-25 11:13:20 UTC, 46.8% complete
2024-07-25 11:04:02,357 INFO [ping_erddap] running in process 773146
2024-07-25 11:04:02,358 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:04:02,360 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-25 11:04:02,361 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:04:02,375 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-25 11:04:02,377 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 11:04:04,025 INFO [make_surface_current_tiles] running in process 773149
2024-07-25 11:04:04,029 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:04:04,030 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-07-25 11:04:04,032 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:04:04,090 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-25 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/25jul24
2024-07-25 11:04:07,706 INFO [make_plots] running in process 773148
2024-07-25 11:04:07,710 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:04:07,725 INFO [make_plots] publishing log messages to tcp://*:5569
2024-07-25 11:04:07,730 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:04:09,340 INFO [make_plots] preparing render list for 2024-07-25 NEMO forecast publish figures
2024-07-25 11:04:09,479 INFO [make_plots] added Website_thumbnail figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,480 INFO [make_plots] added Threshold_website figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,481 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,482 INFO [make_plots] added Boundary Bay figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,483 INFO [make_plots] added Campbell River figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,483 INFO [make_plots] added Cherry Point figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,484 INFO [make_plots] added Friday Harbor figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,485 INFO [make_plots] added Halfmoon Bay figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,486 INFO [make_plots] added Nanaimo figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,487 INFO [make_plots] added Neah Bay figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,488 INFO [make_plots] added New Westminster figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,489 INFO [make_plots] added Patricia Bay figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,489 INFO [make_plots] added Point Atkinson figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,490 INFO [make_plots] added Port Renfrew figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,491 INFO [make_plots] added Sand Heads figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,492 INFO [make_plots] added Sandy Cove figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,493 INFO [make_plots] added Squamish figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,493 INFO [make_plots] added Victoria figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,494 INFO [make_plots] added Woodwards Landing figure to 2024-07-25 NEMO forecast publish render list
2024-07-25 11:04:09,495 INFO [make_plots] starting to render nemo forecast publish 25jul24 figures
2024-07-25 11:04:13,139 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7829044 = 2024-07-25 13:22:00 UTC, 55.7% complete
2024-07-25 11:04:36,958 INFO [download_results] hindcast 2019-02-21 results files from optimum-hindcast downloaded
2024-07-25 11:04:36,968 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 11:04:37,000 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-21'], host='localhost')
2024-07-25 11:04:38,636 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 11:04:52,883 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-25 11:04:52,888 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-25 11:04:53,058 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:04:53,062 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 11:04:54,316 INFO [download_wwatch3_results] running in process 773362
2024-07-25 11:04:54,317 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:04:54,318 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2024-07-25 11:04:54,319 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:04:55,071 INFO [run_ww3] running in process 3885
2024-07-25 11:04:55,072 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:04:55,073 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-07-25 11:04:55,074 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:04:55,076 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-07-25T110455.051018-0700
2024-07-25 11:04:55,084 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-07-25T110455.051018-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-07-25 11:04:55,085 INFO [run_ww3] forecast WaveWatch3 run for 2024-07-25 on arbutus.cloud-nowcast started
2024-07-25 11:04:55,096 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-25 11:04:55,128 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-25 11:04:57,221 INFO [watch_ww3] running in process 4039
2024-07-25 11:04:57,222 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:04:57,222 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-07-25 11:04:57,223 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:04:57,255 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-25 11:06:56,283 INFO [download_wwatch3_results] nowcast 2024-07-25 results files from arbutus.cloud-nowcast downloaded
2024-07-25 11:06:56,286 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-25 11:09:01,636 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17808222 = 2024-07-25 13:07:20 UTC, 54.7% complete
2024-07-25 11:09:13,663 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7829253 = 2024-07-25 15:41:20 UTC, 65.4% complete
2024-07-25 11:09:57,261 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 81, 12.5% complete
2024-07-25 11:14:01,637 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17808393 = 2024-07-25 15:01:20 UTC, 62.6% complete
2024-07-25 11:14:14,149 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7829461 = 2024-07-25 18:00:00 UTC, 75.0% complete
2024-07-25 11:14:36,864 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/25jul24
2024-07-25 11:14:57,263 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 174, 26.9% complete
2024-07-25 11:16:53,053 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-25 11:16:53,054 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-25 11:17:27,811 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:19:01,639 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17808565 = 2024-07-25 16:56:00 UTC, 70.6% complete
2024-07-25 11:19:07,051 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/25jul24
2024-07-25 11:19:07,062 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-25 forecast into /results/nowcast-sys/figures/surface_currents/forecast/25jul24
2024-07-25 11:19:07,063 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-25 forecast completed
2024-07-25 11:19:07,354 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-25 11:19:13,827 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:19:14,188 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:19:14,634 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7829670 = 2024-07-25 20:19:20 UTC, 84.7% complete
2024-07-25 11:19:48,528 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:19:57,263 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 264, 40.7% complete
2024-07-25 11:20:57,163 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:21:32,287 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:22:07,618 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:22:42,506 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:23:17,268 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:23:52,377 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:24:01,640 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17808737 = 2024-07-25 18:50:40 UTC, 78.6% complete
2024-07-25 11:24:15,092 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7829880 = 2024-07-25 22:39:20 UTC, 94.4% complete
2024-07-25 11:24:27,245 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:24:27,604 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:24:57,267 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 354, 54.6% complete
2024-07-25 11:25:02,140 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:25:36,170 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-20 00:00:00Z to 2024-07-27 11:55:00Z
2024-07-25 11:26:11,027 INFO [make_plots] finished rendering nemo forecast publish 25jul24 figures
2024-07-25 11:26:11,029 INFO [make_plots] nemo publish plots for 2024-07-25 forecast completed
2024-07-25 11:26:11,036 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-25 11:26:11,175 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:26:14,091 INFO [make_feeds] running in process 775045
2024-07-25 11:26:14,092 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:26:14,094 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-07-25 11:26:14,095 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:26:41,873 INFO [make_feeds] ATOM feeds for 2024-07-25 forecast run completed
2024-07-25 11:26:41,876 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-25 11:29:01,641 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17808908 = 2024-07-25 20:44:40 UTC, 86.5% complete
2024-07-25 11:29:15,521 INFO [watch_NEMO_agrif] 25jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7830000 = 2024-07-25 23:59:20 UTC, 100.0% complete
2024-07-25 11:29:57,267 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 444, 68.5% complete
2024-07-25 11:34:01,642 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17809080 = 2024-07-25 22:39:20 UTC, 94.4% complete
2024-07-25 11:34:15,742 INFO [watch_NEMO_agrif] NEMO AGRIF run 11101991.orca2.ibb on orcinus-nowcast-agrif completed
2024-07-25 11:34:15,747 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-07-25 11:34:16,116 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:34:17,669 INFO [download_results] running in process 775122
2024-07-25 11:34:17,670 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:34:17,671 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-25 11:34:17,673 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:34:17,674 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/25jul24 to /results/SalishSea/nowcast-agrif.201702
2024-07-25 11:34:40,812 INFO [download_results] nowcast-agrif 2024-07-25 results files from orcinus-nowcast-agrif downloaded
2024-07-25 11:34:40,820 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 11:34:40,992 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:34:44,412 INFO [make_plots] running in process 775129
2024-07-25 11:34:44,414 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:34:44,415 INFO [make_plots] publishing log messages to tcp://*:5564
2024-07-25 11:34:44,416 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:34:44,979 INFO [make_plots] preparing render list for 2024-07-25 NEMO nowcast-agrif research figures
2024-07-25 11:34:44,980 INFO [make_plots] added baynes_sound_surface figure to 2024-07-25 NEMO nowcast-agrif research render list
2024-07-25 11:34:44,981 INFO [make_plots] starting to render nemo nowcast-agrif research 25jul24 figures
2024-07-25 11:34:54,501 INFO [make_plots] finished rendering nemo nowcast-agrif research 25jul24 figures
2024-07-25 11:34:54,503 INFO [make_plots] nemo research plots for 2024-07-25 nowcast-agrif completed
2024-07-25 11:34:54,507 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-25 11:34:57,269 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 534, 82.4% complete
2024-07-25 11:39:01,642 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17809200 = 2024-07-25 23:59:20 UTC, 100.0% complete
2024-07-25 11:39:57,266 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 630, 97.2% complete
2024-07-25 11:44:01,737 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-07-25 11:44:01,740 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-25 11:44:02,059 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:44:03,234 INFO [download_results] running in process 775196
2024-07-25 11:44:03,235 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:44:03,236 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-25 11:44:03,237 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:44:03,238 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/25jul24 to /results2/SalishSea/nowcast-green.202111
2024-07-25 11:44:57,307 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-25 11:44:57,313 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-25 11:44:57,623 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:44:58,674 INFO [download_wwatch3_results] running in process 775206
2024-07-25 11:44:58,675 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:44:58,676 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-07-25 11:44:58,677 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:50:24,085 INFO [download_wwatch3_results] forecast 2024-07-25 results files from arbutus.cloud-nowcast downloaded
2024-07-25 11:50:24,089 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-25 11:50:24,259 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:50:25,593 INFO [update_forecast_datasets] running in process 775229
2024-07-25 11:50:25,595 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:50:25,596 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-07-25 11:50:25,597 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:50:25,598 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/25jul24 run
2024-07-25 11:50:25,599 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/25jul24 run
2024-07-25 11:50:25,853 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/25jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-25 11:50:25,854 INFO [update_forecast_datasets] wwatch3 2024-07-25 forecast rolling forecast datasets updated
2024-07-25 11:50:25,856 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-25 11:50:26,027 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-25 11:50:27,250 INFO [ping_erddap] running in process 775234
2024-07-25 11:50:27,251 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:50:27,252 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-25 11:50:27,254 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:50:27,255 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-25 11:50:27,258 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 11:50:27,429 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 11:50:30,974 INFO [make_plots] running in process 775239
2024-07-25 11:50:30,975 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:50:30,976 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-25 11:50:30,977 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:50:30,978 INFO [make_plots] preparing render list for 2024-07-25 WaveWatch3 forecast publish figures
2024-07-25 11:50:30,979 INFO [make_plots] added Halibut Bank figure to 2024-07-25 WaveWatch3 forecast publish render list
2024-07-25 11:50:30,980 INFO [make_plots] added Sentry Shoal figure to 2024-07-25 WaveWatch3 forecast publish render list
2024-07-25 11:50:30,981 INFO [make_plots] starting to render wwatch3 forecast publish 25jul24 figures
2024-07-25 11:50:31,285 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-25 11:50:38,238 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-25 11:50:44,770 INFO [make_plots] finished rendering wwatch3 forecast publish 25jul24 figures
2024-07-25 11:50:44,771 INFO [make_plots] wwatch3 publish plots for 2024-07-25 forecast completed
2024-07-25 11:50:44,774 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-25 11:58:58,294 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 11:58:58,647 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-02-26'], host='localhost')
2024-07-25 11:58:58,654 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 11:58:58,660 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 11:58:59,858 INFO [download_results] running in process 775295
2024-07-25 11:58:59,860 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:58:59,861 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-25 11:58:59,862 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:58:59,863 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26feb19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 11:59:04,086 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 11:59:30,548 INFO [download_results] hindcast 2019-02-26 results files from optimum-hindcast downloaded
2024-07-25 11:59:30,556 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 11:59:30,728 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-02-26'], host='localhost')
2024-07-25 11:59:31,748 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 11:59:31,922 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-feb', 'robot.graham'], host='localhost')
2024-07-25 11:59:32,911 INFO [archive_tarball] running in process 775338
2024-07-25 11:59:32,912 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 11:59:32,913 INFO [archive_tarball] publishing log messages to tcp://*:5567
2024-07-25 11:59:32,914 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-25 11:59:32,915 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb19.tar from /ocean/sallen/timbucktwo/oxygen/*feb19/
2024-07-25 12:05:54,106 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb19.index from /ocean/dlatorne/oxygen-feb19.tar
2024-07-25 12:05:54,187 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-feb19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-25 12:12:23,068 INFO [archive_tarball] hindcast *feb19 results files archived to robot.graham
2024-07-25 12:12:23,071 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-25 12:21:05,570 INFO [download_results] nowcast-green 2024-07-25 results files from arbutus.cloud-nowcast downloaded
2024-07-25 12:21:05,575 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 12:21:05,746 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 12:21:05,751 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-07-25 12:21:05,755 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 12:21:05,760 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 12:21:05,767 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 12:21:07,177 INFO [ping_erddap] running in process 775655
2024-07-25 12:21:07,178 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 12:21:07,179 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-25 12:21:07,180 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 12:21:07,185 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-07-25 12:21:07,187 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 12:21:08,317 INFO [make_averaged_dataset] running in process 775658
2024-07-25 12:21:08,318 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 12:21:08,320 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2024-07-25 12:21:08,321 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-25 12:21:08,321 INFO [make_averaged_dataset] creating day-averaged dataset for 25-Jul-2024 physics
2024-07-25 12:21:08,325 INFO [make_averaged_dataset] running in process 775656
2024-07-25 12:21:08,326 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 12:21:08,327 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567
2024-07-25 12:21:08,328 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-25 12:21:08,328 INFO [make_averaged_dataset] creating day-averaged dataset for 25-Jul-2024 biology
2024-07-25 12:21:08,330 INFO [make_averaged_dataset] running in process 775657
2024-07-25 12:21:08,331 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 12:21:08,332 INFO [make_averaged_dataset] publishing log messages to tcp://*:5568
2024-07-25 12:21:08,333 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-25 12:21:08,333 INFO [make_averaged_dataset] creating day-averaged dataset for 25-Jul-2024 chemistry
2024-07-25 12:21:09,518 INFO [make_plots] running in process 775653
2024-07-25 12:21:09,519 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 12:21:09,520 INFO [make_plots] publishing log messages to tcp://*:5569
2024-07-25 12:21:09,521 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-25 12:21:10,056 INFO [make_plots] preparing render list for 2024-07-25 NEMO nowcast-green research figures
2024-07-25 12:21:11,739 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:12,445 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:13,194 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:13,892 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:14,663 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:15,380 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:16,096 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:16,910 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:17,710 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:18,514 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:19,314 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:20,038 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:20,734 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:20,735 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:20,735 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:20,736 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-07-25 NEMO nowcast-green research render list
2024-07-25 12:21:20,737 INFO [make_plots] starting to render nemo nowcast-green research 25jul24 figures
2024-07-25 12:21:47,481 INFO [make_averaged_dataset] day-averaged dataset for 25-Jul-2024 physics created
2024-07-25 12:21:47,483 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-25 12:22:30,415 INFO [make_averaged_dataset] day-averaged dataset for 25-Jul-2024 chemistry created
2024-07-25 12:22:30,418 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-25 12:22:45,112 INFO [make_averaged_dataset] day-averaged dataset for 25-Jul-2024 biology created
2024-07-25 12:22:45,116 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-25 13:24:08,513 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 13:24:08,873 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-01'], host='localhost')
2024-07-25 13:24:08,880 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 13:24:08,886 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 13:24:10,451 INFO [download_results] running in process 776097
2024-07-25 13:24:10,452 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 13:24:10,453 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-25 13:24:10,454 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 13:24:10,455 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01mar19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 13:24:14,531 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 13:24:59,478 INFO [download_results] hindcast 2019-03-01 results files from optimum-hindcast downloaded
2024-07-25 13:24:59,491 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 13:24:59,663 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-01'], host='localhost')
2024-07-25 13:25:01,170 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 13:31:59,924 INFO [make_plots] finished rendering nemo nowcast-green research 25jul24 figures
2024-07-25 13:32:00,091 INFO [make_plots] nemo research plots for 2024-07-25 nowcast-green completed
2024-07-25 13:32:00,156 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-25 14:49:19,020 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 14:49:19,381 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-06'], host='localhost')
2024-07-25 14:49:19,388 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 14:49:19,395 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 14:49:20,683 INFO [download_results] running in process 776513
2024-07-25 14:49:20,685 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 14:49:20,686 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-25 14:49:20,687 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 14:49:20,688 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06mar19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 14:49:25,058 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 14:50:01,014 INFO [download_results] hindcast 2019-03-06 results files from optimum-hindcast downloaded
2024-07-25 14:50:01,024 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 14:50:01,229 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-06'], host='localhost')
2024-07-25 14:50:02,349 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 15:15:31,268 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240725/18/
2024-07-25 15:15:31,270 INFO [collect_weather] 2024-07-25 2.5km weather forecast 18 collection complete
2024-07-25 15:15:31,273 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-25 15:15:31,516 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240725/18/
2024-07-25 15:15:31,518 INFO [crop_gribs] 2024-07-25 18 GRIBs cropping complete
2024-07-25 15:15:31,636 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-07-25 15:15:31,644 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-07-25 15:15:31,649 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-07-25 15:15:31,654 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-07-26'], host='localhost')
2024-07-25 15:15:31,661 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-25 15:15:32,858 INFO [download_weather] running in process 776612
2024-07-25 15:15:32,861 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 15:15:32,862 INFO [download_weather] publishing log messages to tcp://*:5562
2024-07-25 15:15:32,863 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-25 15:15:32,864 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240725
2024-07-25 15:15:32,865 INFO [download_weather] running in process 776614
2024-07-25 15:15:32,866 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 15:15:32,867 INFO [download_weather] publishing log messages to tcp://*:5564
2024-07-25 15:15:32,868 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-25 15:15:32,869 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240725
2024-07-25 15:15:32,879 INFO [collect_weather] running in process 776615
2024-07-25 15:15:32,880 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 15:15:32,881 INFO [collect_weather] publishing log messages to tcp://*:5566
2024-07-25 15:15:32,882 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-25 15:15:32,900 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-07-25 15:15:33,826 INFO [crop_gribs] running in process 776616
2024-07-25 15:15:33,827 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 15:15:33,828 INFO [crop_gribs] publishing log messages to tcp://*:5567
2024-07-25 15:15:33,830 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-25 15:15:33,830 INFO [crop_gribs] cropping 2024-07-26 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-07-25 15:15:33,843 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240726/00/
2024-07-25 15:20:30,346 INFO [download_weather] 2024-07-25 1km weather forecast 00 downloads complete
2024-07-25 15:20:30,349 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-25 15:20:35,881 INFO [download_weather] 2024-07-25 1km weather forecast 12 downloads complete
2024-07-25 15:20:35,883 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-25 16:19:29,109 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 16:19:29,455 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-11'], host='localhost')
2024-07-25 16:19:29,461 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 16:19:29,466 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 16:19:30,677 INFO [download_results] running in process 776868
2024-07-25 16:19:30,678 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 16:19:30,679 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-25 16:19:30,680 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 16:19:30,682 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11mar19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 16:19:34,905 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 16:20:10,754 INFO [download_results] hindcast 2019-03-11 results files from optimum-hindcast downloaded
2024-07-25 16:20:10,764 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 16:20:10,968 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-11'], host='localhost')
2024-07-25 16:20:12,177 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 17:44:39,486 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 17:44:39,841 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-16'], host='localhost')
2024-07-25 17:44:39,848 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 17:44:39,854 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 17:44:41,093 INFO [download_results] running in process 777386
2024-07-25 17:44:41,094 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 17:44:41,095 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-25 17:44:41,096 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 17:44:41,097 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16mar19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 17:44:45,371 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 17:45:21,646 INFO [download_results] hindcast 2019-03-16 results files from optimum-hindcast downloaded
2024-07-25 17:45:21,656 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 17:45:21,857 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-16'], host='localhost')
2024-07-25 17:45:23,004 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 19:09:49,740 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 19:09:50,091 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-21'], host='localhost')
2024-07-25 19:09:50,099 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 19:09:50,104 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 19:09:51,363 INFO [download_results] running in process 777777
2024-07-25 19:09:51,364 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 19:09:51,365 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-25 19:09:51,367 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 19:09:51,368 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21mar19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 19:09:55,615 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 19:10:31,618 INFO [download_results] hindcast 2019-03-21 results files from optimum-hindcast downloaded
2024-07-25 19:10:31,627 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 19:10:31,828 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-21'], host='localhost')
2024-07-25 19:10:33,217 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 20:55:01,741 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 20:55:02,144 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-03-26'], host='localhost')
2024-07-25 20:55:02,151 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 20:55:02,156 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 20:55:03,408 INFO [download_results] running in process 778128
2024-07-25 20:55:03,409 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 20:55:03,410 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-25 20:55:03,411 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 20:55:03,413 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26mar19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 20:55:07,530 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 20:55:49,159 INFO [download_results] hindcast 2019-03-26 results files from optimum-hindcast downloaded
2024-07-25 20:55:49,171 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 20:55:49,374 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-03-26'], host='localhost')
2024-07-25 20:55:50,834 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 20:55:51,040 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-mar', 'robot.graham'], host='localhost')
2024-07-25 20:55:52,276 INFO [archive_tarball] running in process 778213
2024-07-25 20:55:52,277 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 20:55:52,278 INFO [archive_tarball] publishing log messages to tcp://*:5564
2024-07-25 20:55:52,279 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-25 20:55:52,280 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar19.tar from /ocean/sallen/timbucktwo/oxygen/*mar19/
2024-07-25 21:01:41,169 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar19.index from /ocean/dlatorne/oxygen-mar19.tar
2024-07-25 21:01:41,254 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-mar19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-25 21:19:41,676 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240726/00/
2024-07-25 21:19:41,677 INFO [collect_weather] 2024-07-26 2.5km weather forecast 00 collection complete
2024-07-25 21:19:41,680 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-25 21:19:42,022 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-07-25 21:19:42,028 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-07-26'], host='localhost')
2024-07-25 21:19:43,377 INFO [collect_weather] running in process 778317
2024-07-25 21:19:43,378 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 21:19:43,379 INFO [collect_weather] publishing log messages to tcp://*:5565
2024-07-25 21:19:43,380 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-25 21:19:43,396 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-07-25 21:19:43,508 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240726/00/
2024-07-25 21:19:43,511 INFO [crop_gribs] 2024-07-26 00 GRIBs cropping complete
2024-07-25 21:19:43,513 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-25 21:19:44,178 INFO [crop_gribs] running in process 778319
2024-07-25 21:19:44,179 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 21:19:44,181 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-07-25 21:19:44,182 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-25 21:19:44,183 INFO [crop_gribs] cropping 2024-07-26 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-07-25 21:19:44,198 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240726/06/
2024-07-25 21:22:04,207 INFO [archive_tarball] hindcast *mar19 results files archived to robot.graham
2024-07-25 21:22:04,210 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-25 22:20:12,154 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 22:20:12,511 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-01'], host='localhost')
2024-07-25 22:20:12,517 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 22:20:12,524 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 22:20:13,803 INFO [download_results] running in process 778704
2024-07-25 22:20:13,804 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 22:20:13,806 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-25 22:20:13,807 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 22:20:13,808 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01apr19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 22:20:17,972 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 22:20:53,933 INFO [download_results] hindcast 2019-04-01 results files from optimum-hindcast downloaded
2024-07-25 22:20:53,944 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 22:20:54,147 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-01'], host='localhost')
2024-07-25 22:20:55,375 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 23:50:22,615 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 23:50:22,979 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-06'], host='localhost')
2024-07-25 23:50:22,986 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 23:50:22,992 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 23:50:24,232 INFO [download_results] running in process 779023
2024-07-25 23:50:24,234 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 23:50:24,235 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-25 23:50:24,236 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 23:50:24,237 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06apr19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 23:50:28,421 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 23:51:04,532 INFO [download_results] hindcast 2019-04-06 results files from optimum-hindcast downloaded
2024-07-25 23:51:04,542 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 23:51:04,746 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-06'], host='localhost')
2024-07-25 23:51:06,141 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-26 01:15:32,886 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-26 01:15:33,243 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-11'], host='localhost')
2024-07-26 01:15:33,250 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-26 01:15:33,254 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-26 01:15:34,485 INFO [download_results] running in process 779397
2024-07-26 01:15:34,486 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 01:15:34,488 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-26 01:15:34,489 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-26 01:15:34,490 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11apr19 to /ocean/sallen/timbucktwo/oxygen
2024-07-26 01:15:41,273 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-26 01:16:14,535 INFO [download_results] hindcast 2019-04-11 results files from optimum-hindcast downloaded
2024-07-26 01:16:14,546 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-26 01:16:14,750 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-11'], host='localhost')
2024-07-26 01:16:15,965 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-26 02:45:43,466 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-26 02:45:43,815 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-16'], host='localhost')
2024-07-26 02:45:43,822 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-26 02:45:43,828 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-26 02:45:45,099 INFO [download_results] running in process 779831
2024-07-26 02:45:45,100 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 02:45:45,102 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-26 02:45:45,103 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-26 02:45:45,104 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16apr19 to /ocean/sallen/timbucktwo/oxygen
2024-07-26 02:45:49,508 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-26 02:46:25,795 INFO [download_results] hindcast 2019-04-16 results files from optimum-hindcast downloaded
2024-07-26 02:46:25,804 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-26 02:46:26,010 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-16'], host='localhost')
2024-07-26 02:46:27,266 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-26 03:14:31,089 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240726/06/
2024-07-26 03:14:31,090 INFO [collect_weather] 2024-07-26 2.5km weather forecast 06 collection complete
2024-07-26 03:14:31,094 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-26 03:14:31,372 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240726/06/
2024-07-26 03:14:31,373 INFO [crop_gribs] 2024-07-26 06 GRIBs cropping complete
2024-07-26 03:14:31,440 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,447 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,452 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,457 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,461 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,466 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,470 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,474 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,481 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,486 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,490 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,495 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,511 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,531 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,560 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-07-26 03:14:31,572 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-07-26 03:14:31,579 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-07-26 03:14:31,587 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-07-25'], host='localhost')
2024-07-26 03:14:31,594 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-07-26 03:14:31,602 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-07-26 03:14:31,609 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-07-26 03:14:31,620 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-26 03:14:32,249 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-07-26 03:14:34,602 INFO [collect_NeahBay_ssh] running in process 780063
2024-07-26 03:14:34,603 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:34,604 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2024-07-26 03:14:34,606 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:34,606 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-07-26
2024-07-26 03:14:34,637 INFO [collect_weather] running in process 780064
2024-07-26 03:14:34,642 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:34,644 INFO [collect_weather] publishing log messages to tcp://*:5565
2024-07-26 03:14:34,645 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:34,676 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-07-26 03:14:35,468 INFO [collect_river_data] running in process 780049
2024-07-26 03:14:35,469 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,470 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-07-26 03:14:35,472 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,473 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-07-25
2024-07-26 03:14:35,503 INFO [collect_river_data] running in process 780055
2024-07-26 03:14:35,505 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,506 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-26 03:14:35,507 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,508 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-07-25
2024-07-26 03:14:35,564 INFO [collect_river_data] running in process 780044
2024-07-26 03:14:35,565 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,566 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-26 03:14:35,567 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,567 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-07-25
2024-07-26 03:14:35,576 INFO [collect_river_data] running in process 780051
2024-07-26 03:14:35,577 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,578 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-26 03:14:35,578 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,579 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-07-25
2024-07-26 03:14:35,616 INFO [collect_river_data] running in process 780048
2024-07-26 03:14:35,617 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,618 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-26 03:14:35,620 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,620 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-07-25
2024-07-26 03:14:35,684 INFO [collect_river_data] running in process 780053
2024-07-26 03:14:35,684 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,685 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-26 03:14:35,687 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,688 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-07-25
2024-07-26 03:14:35,689 INFO [collect_river_data] running in process 780054
2024-07-26 03:14:35,690 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,691 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-07-26 03:14:35,692 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,693 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-07-25
2024-07-26 03:14:35,712 INFO [collect_river_data] running in process 780052
2024-07-26 03:14:35,713 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,714 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-07-26 03:14:35,715 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,716 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-07-25
2024-07-26 03:14:35,717 INFO [collect_river_data] running in process 780047
2024-07-26 03:14:35,718 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,719 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-07-26 03:14:35,720 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,721 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-07-25
2024-07-26 03:14:35,730 INFO [collect_river_data] running in process 780057
2024-07-26 03:14:35,731 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,732 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-07-26 03:14:35,732 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,734 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-07-25
2024-07-26 03:14:35,780 INFO [collect_river_data] running in process 780058
2024-07-26 03:14:35,781 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,782 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-07-26 03:14:35,783 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,784 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-07-25
2024-07-26 03:14:35,788 INFO [collect_river_data] running in process 780046
2024-07-26 03:14:35,791 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,792 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-07-26 03:14:35,792 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,793 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-07-25
2024-07-26 03:14:35,812 INFO [collect_river_data] running in process 780050
2024-07-26 03:14:35,813 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,814 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-07-26 03:14:35,815 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,815 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-07-25
2024-07-26 03:14:35,856 INFO [collect_river_data] running in process 780056
2024-07-26 03:14:35,857 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:35,858 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-07-26 03:14:35,859 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:35,860 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-07-25
2024-07-26 03:14:36,052 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-07-25 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-07-26 03:14:36,053 INFO [collect_river_data] ECCC Fraser river data collection for 2024-07-25 completed
2024-07-26 03:14:36,056 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:36,143 INFO [collect_NeahBay_ssh] 2024-07-26T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-07-26 03:14:36,205 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-07-26 03:14:36,207 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-07-25 completed
2024-07-26 03:14:36,254 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-07-25 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-07-26 03:14:36,255 INFO [collect_river_data] ECCC Capilano river data collection for 2024-07-25 completed
2024-07-26 03:14:36,302 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-07-26 03:14:36,303 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-07-25 completed
2024-07-26 03:14:36,306 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-07-25 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-07-26 03:14:36,307 INFO [collect_river_data] ECCC Englishman river data collection for 2024-07-25 completed
2024-07-26 03:14:36,330 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-07-26 03:14:36,331 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-07-25 completed
2024-07-26 03:14:36,386 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-07-26 03:14:36,387 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-07-25 completed
2024-07-26 03:14:36,403 INFO [get_vfpa_hadcp] running in process 780062
2024-07-26 03:14:36,404 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:36,406 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5590
2024-07-26 03:14:36,406 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:36,407 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-25
2024-07-26 03:14:36,409 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-07-26 03:14:36,409 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-07-25 completed
2024-07-26 03:14:36,464 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-07-26 03:14:36,465 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-07-25 completed
2024-07-26 03:14:36,515 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-07-26 03:14:36,516 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-07-25 completed
2024-07-26 03:14:36,529 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-26 03:14:36,546 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-07-26 03:14:36,547 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-07-25 completed
2024-07-26 03:14:36,551 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-07-26 03:14:36,552 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-07-25 completed
2024-07-26 03:14:36,557 INFO [grib_to_netcdf] running in process 780121
2024-07-26 03:14:36,558 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:36,559 INFO [grib_to_netcdf] publishing log messages to tcp://*:5591
2024-07-26 03:14:36,560 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:36,560 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-26 forecast2 run
2024-07-26 03:14:36,564 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-07-26 03:14:36,564 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-07-25 completed
2024-07-26 03:14:36,577 INFO [crop_gribs] running in process 780065
2024-07-26 03:14:36,578 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:36,579 INFO [crop_gribs] publishing log messages to tcp://*:5592
2024-07-26 03:14:36,580 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:36,580 INFO [crop_gribs] cropping 2024-07-26 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-07-26 03:14:36,591 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240726/12/
2024-07-26 03:14:36,604 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-07-25 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-07-26 03:14:36,605 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-07-25 completed
2024-07-26 03:14:36,731 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-07-26'], host='localhost')
2024-07-26 03:14:36,736 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:36,836 INFO [get_onc_ctd] running in process 780059
2024-07-26 03:14:36,837 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:36,838 INFO [get_onc_ctd] publishing log messages to tcp://*:5566
2024-07-26 03:14:36,839 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:36,840 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-07-25
2024-07-26 03:14:36,869 INFO [get_onc_ferry] running in process 780061
2024-07-26 03:14:36,869 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:36,870 INFO [get_onc_ferry] publishing log messages to tcp://*:5593
2024-07-26 03:14:36,871 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:36,871 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-07-25
2024-07-26 03:14:36,895 INFO [get_onc_ctd] running in process 780060
2024-07-26 03:14:36,897 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:36,898 INFO [get_onc_ctd] publishing log messages to tcp://*:5594
2024-07-26 03:14:36,899 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:36,900 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-07-25
2024-07-26 03:14:36,960 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:37,173 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:37,364 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:37,553 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:37,741 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:37,950 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:38,164 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:38,358 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:38,577 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:38,601 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-25 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202407.nc
2024-07-26 03:14:38,602 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-07 netcdf file
2024-07-26 03:14:38,769 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:38,976 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-07-25
2024-07-26 03:14:38,978 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:39,182 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-26 03:14:39,376 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2024-07-26 03:14:39,390 CRITICAL [get_onc_ctd] 2024-07-25 ONC SCVIP CTD T&S file creation failed
2024-07-26 03:14:39,650 INFO [make_ssh_files] running in process 780501
2024-07-26 03:14:39,651 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2024-07-26 03:14:39,653 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:39,655 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2024-07-26 03:14:39,657 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:39,658 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-07-26 NOAA Neah Bay 00Z observation and forecast values
2024-07-26 03:14:40,745 INFO [ping_erddap] running in process 780545
2024-07-26 03:14:40,746 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:40,747 INFO [ping_erddap] publishing log messages to tcp://*:5566
2024-07-26 03:14:40,748 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:40,750 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2024-07-26 03:14:40,751 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-26 03:14:44,142 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-26 03:14:44,328 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d22.nc
2024-07-26 03:14:44,510 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d23.nc
2024-07-26 03:14:44,692 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d24.nc
2024-07-26 03:14:44,874 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d25.nc
2024-07-26 03:14:44,941 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d27.nc
2024-07-26 03:14:45,076 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d26.nc
2024-07-26 03:14:45,253 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d27.nc
2024-07-26 03:14:45,450 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d28.nc
2024-07-26 03:14:45,622 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d29.nc
2024-07-26 03:14:45,744 INFO [make_ssh_files] sea surface height boundary file for 2024-07-26 forecast2 run created
2024-07-26 03:14:45,747 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-26 03:14:45,952 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-26 03:14:45,957 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-26 03:14:47,356 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d28.nc
2024-07-26 03:14:47,362 INFO [grib_to_netcdf] 2024-07-26 NEMO-atmos forcing file for forecast2 created
2024-07-26 03:14:47,365 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-26 03:14:47,414 INFO [make_runoff_file] running in process 780575
2024-07-26 03:14:47,416 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:47,417 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2024-07-26 03:14:47,417 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:47,442 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-26 03:14:47,522 INFO [make_runoff_file] 2024-07-26 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-26 03:14:47,579 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-26 03:14:47,953 INFO [make_v202111_runoff_file] running in process 780573
2024-07-26 03:14:47,954 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:47,955 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5566
2024-07-26 03:14:47,956 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:47,958 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-25
2024-07-26 03:14:48,821 ERROR [make_v202111_runoff_file] no 2024-07-25 discharge obs for Skagit_MountVernon: patching
2024-07-26 03:14:49,346 ERROR [make_v202111_runoff_file] no 2024-07-25 discharge obs for Snohomish_Monroe: patching
2024-07-26 03:14:49,550 ERROR [make_v202111_runoff_file] no 2024-07-25 discharge obs for Nisqually_McKenna: patching
2024-07-26 03:14:49,636 ERROR [make_v202111_runoff_file] no 2024-07-25 discharge obs for Greenwater_Greenwater: patching
2024-07-26 03:14:50,898 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-25: /results/forcing/rivers/R202108Dailies_y2024m07d25.nc
2024-07-26 03:14:50,899 INFO [make_v202111_runoff_file] 2024-07-25 runoff file creation completed
2024-07-26 03:14:50,901 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-26 03:14:51,075 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-07-26 03:14:51,078 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-07-26 03:14:51,082 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-07-26 03:14:51,087 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-07-26 03:14:52,289 INFO [upload_forcing] running in process 780617
2024-07-26 03:14:52,290 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:52,291 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-07-26 03:14:52,292 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:52,294 INFO [upload_forcing] running in process 780614
2024-07-26 03:14:52,295 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:52,296 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-26 03:14:52,298 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:52,299 INFO [upload_forcing] running in process 780616
2024-07-26 03:14:52,300 INFO [upload_forcing] running in process 780615
2024-07-26 03:14:52,301 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:52,302 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:52,303 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-26 03:14:52,305 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-26 03:14:52,306 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:52,306 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:53,607 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d25.nc
2024-07-26 03:14:53,641 INFO [upload_forcing] forecast2 2024-07-26 forcing files upload to optimum-hindcast completed
2024-07-26 03:14:53,656 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-26 03:14:53,948 INFO [upload_forcing] forecast2 2024-07-26 forcing files upload to orcinus-nowcast-agrif completed
2024-07-26 03:14:53,951 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-26 03:14:54,764 INFO [upload_forcing] forecast2 2024-07-26 forcing files upload to arbutus.cloud-nowcast completed
2024-07-26 03:14:54,767 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-26 03:14:54,988 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-26'], host='localhost')
2024-07-26 03:14:55,987 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-07-25
2024-07-26 03:14:56,304 INFO [make_forcing_links] running in process 780644
2024-07-26 03:14:56,305 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:56,307 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-07-26 03:14:56,308 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:56,820 WARNING [get_onc_ferry] No ONC TWDP TSG temperature,conductivity,salinity data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:56,825 WARNING [get_onc_ferry] No ONC TWDP TSG temperature data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:56,827 WARNING [get_onc_ferry] No ONC TWDP TSG conductivity data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:56,828 WARNING [get_onc_ferry] No ONC TWDP TSG salinity data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:56,830 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-07-25
2024-07-26 03:14:57,453 INFO [make_forcing_links] forecast2 2024-07-26 forcing file links on arbutus.cloud-nowcast created
2024-07-26 03:14:57,457 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-26 03:14:57,663 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-26'], host='arbutus.cloud-nowcast')
2024-07-26 03:14:57,794 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation,oxygen_corrected,temperature data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:57,799 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:57,801 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:57,802 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR temperature data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:57,803 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-07-25
2024-07-26 03:14:57,898 WARNING [get_onc_ctd] no SEVIP salinity data
2024-07-26 03:14:58,187 INFO [get_onc_ctd] 2024-07-25 ONC SEVIP CTD T&S file created
2024-07-26 03:14:58,190 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-07-26 03:14:58,398 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-07-26 03:14:58,754 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence,chlorophyll,turbidity data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:58,760 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:58,761 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL chlorophyll data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:58,763 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL turbidity data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:58,763 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-07-25
2024-07-26 03:14:59,551 INFO [ping_erddap] running in process 780653
2024-07-26 03:14:59,553 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:59,554 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-26 03:14:59,555 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:59,557 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-07-26 03:14:59,559 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-26 03:14:59,753 INFO [run_NEMO] running in process 10301
2024-07-26 03:14:59,754 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:14:59,755 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-26 03:14:59,824 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:14:59,851 INFO [upload_forcing] forecast2 2024-07-26 forcing files upload to robot.graham completed
2024-07-26 03:14:59,856 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-26 03:14:59,981 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:59,986 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:59,988 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2024-07-25; substituting empty dataset
2024-07-26 03:14:59,989 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-07-25
2024-07-26 03:15:00,730 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/25jul24forecast2_2024-07-26T031459.889155-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-26 03:15:00,750 INFO [run_NEMO] forecast2 NEMO run for 2024-07-26 on arbutus.cloud-nowcast started
2024-07-26 03:15:00,755 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-26 03:15:00,926 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-26 03:15:02,902 INFO [watch_NEMO] running in process 10553
2024-07-26 03:15:02,903 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:15:02,904 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-26 03:15:02,905 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:15:02,975 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-26 03:15:17,789 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-07-25
2024-07-26 03:15:26,314 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-07-25
2024-07-26 03:15:34,716 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-07-25; substituting empty dataset
2024-07-26 03:15:34,717 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-07-25
2024-07-26 03:15:42,720 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-25; substituting empty dataset
2024-07-26 03:15:42,722 WARNING [get_onc_ferry] ONC TWDP TSG temperature data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,742 WARNING [get_onc_ferry] ONC TWDP TSG conductivity data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,805 WARNING [get_onc_ferry] ONC TWDP TSG salinity data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,822 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,839 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,856 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR temperature data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,873 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,890 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL chlorophyll data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,908 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL turbidity data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,925 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:42,943 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:43,044 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:15:43,061 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-25 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-26 03:16:57,011 INFO [get_onc_ferry] 2024-07-25 ONC TWDP ferry data file created
2024-07-26 03:16:57,014 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-07-26 03:16:57,217 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-07-26 03:16:58,348 INFO [ping_erddap] running in process 780668
2024-07-26 03:16:58,349 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:16:58,351 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-26 03:16:58,352 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:16:58,354 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-07-26 03:16:58,356 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-26 03:20:02,976 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17811803 = 2024-07-27 04:54:40 UTC, 16.4% complete
2024-07-26 03:25:02,975 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17812288 = 2024-07-27 10:18:00 UTC, 34.3% complete
2024-07-26 03:30:02,976 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17812768 = 2024-07-27 15:38:00 UTC, 52.1% complete
2024-07-26 03:35:02,977 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17813245 = 2024-07-27 20:56:00 UTC, 69.8% complete
2024-07-26 03:40:02,978 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17813703 = 2024-07-28 02:01:20 UTC, 86.8% complete
2024-07-26 03:45:02,979 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17814060 = 2024-07-28 05:59:20 UTC, 100.0% complete
2024-07-26 03:50:02,984 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-07-26 03:50:02,988 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-26 03:50:03,348 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-26'], host='arbutus.cloud-nowcast')
2024-07-26 03:50:03,355 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-26'], host='arbutus.cloud-nowcast')
2024-07-26 03:50:03,360 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-26 03:50:04,934 INFO [download_results] running in process 780785
2024-07-26 03:50:04,935 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:50:04,937 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-26 03:50:04,938 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:50:04,939 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/25jul24 to /results/SalishSea/forecast2.202111
2024-07-26 03:50:06,288 INFO [make_ww3_current_file] running in process 11063
2024-07-26 03:50:06,290 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:50:06,291 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-26 03:50:06,292 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:50:06,294 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-26 forecast2 run
2024-07-26 03:50:06,295 INFO [make_ww3_wind_file] running in process 11061
2024-07-26 03:50:06,297 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 03:50:06,298 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-26 03:50:06,299 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-26 03:50:06,300 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-26 forecast2 run
2024-07-26 03:50:15,952 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-26 forecast2 run
2024-07-26 03:50:15,957 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-26 03:52:15,783 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-26 forecast2 run
2024-07-26 03:52:15,788 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-26 03:52:16,003 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-26'], host='arbutus.cloud-nowcast')
2024-07-26 03:52:19,786 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-26 03:52:19,995 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-26 03:52:23,971 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-26 03:57:24,194 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 81, 15.0% complete
2024-07-26 04:02:24,426 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 171, 31.7% complete
2024-07-26 04:07:24,374 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 264, 48.9% complete
2024-07-26 04:10:53,755 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-26 04:10:54,103 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-04-21'], host='localhost')
2024-07-26 04:10:54,110 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-26 04:10:54,115 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-26 04:10:55,349 INFO [download_results] running in process 780837
2024-07-26 04:10:55,350 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:10:55,351 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-26 04:10:55,352 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:10:55,353 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21apr19 to /ocean/sallen/timbucktwo/oxygen
2024-07-26 04:10:59,767 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-26 04:11:35,777 INFO [download_results] hindcast 2019-04-21 results files from optimum-hindcast downloaded
2024-07-26 04:11:35,787 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-26 04:11:35,962 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-04-21'], host='localhost')
2024-07-26 04:11:37,335 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-26 04:12:24,418 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 363, 67.2% complete
2024-07-26 04:14:53,424 INFO [download_results] forecast2 2024-07-25 results files from arbutus.cloud-nowcast downloaded
2024-07-26 04:14:53,428 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-26 04:14:53,632 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-26 04:14:55,984 INFO [make_CHS_currents_file] running in process 780887
2024-07-26 04:14:55,986 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:14:55,987 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-07-26 04:14:55,988 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:15:30,032 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-25 for forecast2
2024-07-26 04:15:30,035 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-26 04:15:30,234 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-26 04:15:31,411 INFO [update_forecast_datasets] running in process 780914
2024-07-26 04:15:31,412 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:15:31,413 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-07-26 04:15:31,414 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:15:31,415 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/25jul24 run
2024-07-26 04:17:23,840 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 462, 85.6% complete
2024-07-26 04:19:42,480 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/25jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-26 04:19:42,481 INFO [update_forecast_datasets] nemo 2024-07-25 forecast2 rolling forecast datasets updated
2024-07-26 04:19:42,484 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-26 04:19:42,686 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-26 04:19:42,692 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-07-25'], host='localhost')
2024-07-26 04:19:42,697 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-26 04:19:43,936 INFO [ping_erddap] running in process 781027
2024-07-26 04:19:43,939 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:19:43,940 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-26 04:19:43,940 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:19:43,950 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-26 04:19:43,951 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-26 04:19:45,174 INFO [make_surface_current_tiles] running in process 781030
2024-07-26 04:19:45,175 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:19:45,177 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2024-07-26 04:19:45,178 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:19:45,179 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-25 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/25jul24
2024-07-26 04:19:47,114 INFO [make_plots] running in process 781029
2024-07-26 04:19:47,115 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:19:47,116 INFO [make_plots] publishing log messages to tcp://*:5564
2024-07-26 04:19:47,117 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:19:47,744 INFO [make_plots] preparing render list for 2024-07-25 NEMO forecast2 publish figures
2024-07-26 04:19:47,864 INFO [make_plots] added Website_thumbnail figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,865 INFO [make_plots] added Threshold_website figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,866 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,867 INFO [make_plots] added Boundary Bay figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,868 INFO [make_plots] added Campbell River figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,869 INFO [make_plots] added Cherry Point figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,870 INFO [make_plots] added Friday Harbor figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,870 INFO [make_plots] added Halfmoon Bay figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,871 INFO [make_plots] added Nanaimo figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,872 INFO [make_plots] added Neah Bay figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,873 INFO [make_plots] added New Westminster figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,873 INFO [make_plots] added Patricia Bay figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,874 INFO [make_plots] added Point Atkinson figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,875 INFO [make_plots] added Port Renfrew figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,876 INFO [make_plots] added Sand Heads figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,876 INFO [make_plots] added Sandy Cove figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,877 INFO [make_plots] added Squamish figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,878 INFO [make_plots] added Victoria figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,878 INFO [make_plots] added Woodwards Landing figure to 2024-07-25 NEMO forecast2 publish render list
2024-07-26 04:19:47,879 INFO [make_plots] starting to render nemo forecast2 publish 25jul24 figures
2024-07-26 04:22:23,838 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-26 04:27:23,839 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-26 04:27:23,841 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-26 04:27:24,267 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-26'], host='localhost')
2024-07-26 04:27:25,829 INFO [download_wwatch3_results] running in process 782438
2024-07-26 04:27:25,830 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:27:25,831 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-07-26 04:27:25,832 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:29:43,913 INFO [download_wwatch3_results] forecast2 2024-07-26 results files from arbutus.cloud-nowcast downloaded
2024-07-26 04:29:43,916 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-26 04:29:44,329 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-07-26'], host='localhost')
2024-07-26 04:29:46,025 INFO [update_forecast_datasets] running in process 782845
2024-07-26 04:29:46,026 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:29:46,027 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566
2024-07-26 04:29:46,028 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:29:46,029 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/26jul24 run
2024-07-26 04:29:46,030 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/26jul24 run
2024-07-26 04:30:18,692 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/26jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-26 04:30:18,693 INFO [update_forecast_datasets] wwatch3 2024-07-26 forecast2 rolling forecast datasets updated
2024-07-26 04:30:18,696 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-26 04:30:19,087 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-26 04:30:20,586 INFO [ping_erddap] running in process 782954
2024-07-26 04:30:20,587 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:30:20,588 INFO [ping_erddap] publishing log messages to tcp://*:5566
2024-07-26 04:30:20,589 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:30:20,590 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-26 04:30:20,591 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-26 04:30:20,996 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-07-26'], host='localhost')
2024-07-26 04:30:26,346 INFO [make_plots] running in process 782964
2024-07-26 04:30:26,347 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:30:26,348 INFO [make_plots] publishing log messages to tcp://*:5566
2024-07-26 04:30:26,349 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:30:26,350 INFO [make_plots] preparing render list for 2024-07-26 WaveWatch3 forecast2 publish figures
2024-07-26 04:30:26,350 INFO [make_plots] added Halibut Bank figure to 2024-07-26 WaveWatch3 forecast2 publish render list
2024-07-26 04:30:26,351 INFO [make_plots] added Sentry Shoal figure to 2024-07-26 WaveWatch3 forecast2 publish render list
2024-07-26 04:30:26,352 INFO [make_plots] starting to render wwatch3 forecast2 publish 26jul24 figures
2024-07-26 04:30:26,830 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-26 04:30:34,142 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-26 04:30:40,680 INFO [make_plots] finished rendering wwatch3 forecast2 publish 26jul24 figures
2024-07-26 04:30:40,681 INFO [make_plots] wwatch3 publish plots for 2024-07-26 forecast2 completed
2024-07-26 04:30:40,683 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-26 04:31:00,164 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/25jul24
2024-07-26 04:33:15,680 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-26 04:33:15,681 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-26 04:33:50,955 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:35:10,602 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/25jul24
2024-07-26 04:35:10,612 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-25 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/25jul24
2024-07-26 04:35:10,613 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-25 forecast2 completed
2024-07-26 04:35:10,911 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-26 04:35:38,007 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:35:38,358 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:36:13,404 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:37:22,986 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:37:58,788 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:38:34,315 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:39:10,325 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:39:44,354 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:40:19,733 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:40:54,905 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:40:55,255 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:41:30,162 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:42:05,175 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-21 00:00:00Z to 2024-07-28 05:55:00Z
2024-07-26 04:42:40,485 INFO [make_plots] finished rendering nemo forecast2 publish 25jul24 figures
2024-07-26 04:42:40,488 INFO [make_plots] nemo publish plots for 2024-07-25 forecast2 completed
2024-07-26 04:42:40,495 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-26 04:42:40,811 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-26 04:42:43,653 INFO [make_feeds] running in process 783086
2024-07-26 04:42:43,655 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:42:43,656 INFO [make_feeds] publishing log messages to tcp://*:5562
2024-07-26 04:42:43,657 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:43:12,767 INFO [make_feeds] ATOM feeds for 2024-07-25 forecast2 run completed
2024-07-26 04:43:12,771 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-26 04:43:13,091 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-07-26 04:43:14,311 INFO [clear_checklist] running in process 783110
2024-07-26 04:43:14,313 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:43:14,314 INFO [clear_checklist] publishing log messages to tcp://*:5562
2024-07-26 04:43:14,315 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:43:14,317 INFO [clear_checklist] requesting that manager clear system state checklist
2024-07-26 04:43:14,319 INFO [manager] writing checklist to log file
2024-07-26 04:43:14,367 INFO [manager] checklist cleared
2024-07-26 04:43:14,370 INFO [clear_checklist] nowcast system checklist cleared
2024-07-26 04:43:14,373 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-07-26 04:43:15,629 INFO [rotate_logs] running in process 783115
2024-07-26 04:43:15,631 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-26 04:43:15,632 INFO [rotate_logs] publishing log messages to tcp://*:5562
2024-07-26 04:43:15,633 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-07-26 04:43:15,634 INFO [rotate_logs] rotating log files