2024-07-24 04:28:07,285 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-07-24 04:28:07,293 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-07-24 04:28:07,298 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-07-24 04:28:07,299 INFO [rotate_logs] log files rotated
2024-07-24 04:28:07,304 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-07-24 05:35:23,018 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 05:35:23,182 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-11'], host='localhost')
2024-07-24 05:35:23,189 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 05:35:23,194 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 05:35:24,420 INFO [download_results] running in process 754450
2024-07-24 05:35:24,421 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 05:35:24,422 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-24 05:35:24,424 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 05:35:24,425 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 05:35:28,678 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 05:36:04,321 INFO [download_results] hindcast 2018-11-11 results files from optimum-hindcast downloaded
2024-07-24 05:36:04,332 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 05:36:04,337 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-11'], host='localhost')
2024-07-24 05:36:05,552 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 07:00:33,042 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 07:00:33,220 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-16'], host='localhost')
2024-07-24 07:00:33,228 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 07:00:33,233 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 07:00:34,482 INFO [download_results] running in process 754929
2024-07-24 07:00:34,483 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 07:00:34,484 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-24 07:00:34,485 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 07:00:34,487 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 07:00:38,783 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 07:01:14,146 INFO [download_results] hindcast 2018-11-16 results files from optimum-hindcast downloaded
2024-07-24 07:01:14,156 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 07:01:14,163 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-16'], host='localhost')
2024-07-24 07:01:15,348 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 08:30:43,362 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 08:30:43,531 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-21'], host='localhost')
2024-07-24 08:30:43,538 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 08:30:43,542 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 08:30:44,796 INFO [download_results] running in process 755557
2024-07-24 08:30:44,797 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 08:30:44,798 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-24 08:30:44,799 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 08:30:44,800 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 08:30:49,011 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 08:31:24,619 INFO [download_results] hindcast 2018-11-21 results files from optimum-hindcast downloaded
2024-07-24 08:31:24,630 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 08:31:24,638 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-21'], host='localhost')
2024-07-24 08:31:25,791 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 09:16:29,844 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240724/12/
2024-07-24 09:16:29,846 INFO [collect_weather] 2024-07-24 2.5km weather forecast 12 collection complete
2024-07-24 09:16:29,848 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-24 09:16:30,016 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 09:16:30,023 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 09:16:30,029 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 09:16:30,035 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 09:16:30,041 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-07-24 09:16:30,047 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-07-24 09:16:30,052 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-07-24 09:16:30,059 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-07-24 09:16:30,067 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-07-24 09:16:31,739 INFO [collect_NeahBay_ssh] running in process 755856
2024-07-24 09:16:31,740 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:31,741 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2024-07-24 09:16:31,742 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:31,743 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-07-24
2024-07-24 09:16:31,752 INFO [collect_weather] running in process 755858
2024-07-24 09:16:31,753 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:31,755 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-07-24 09:16:31,757 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:31,767 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-07-24 09:16:31,848 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240724/12/
2024-07-24 09:16:31,849 INFO [crop_gribs] 2024-07-24 12 GRIBs cropping complete
2024-07-24 09:16:31,851 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-24 09:16:32,014 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-07-24 09:16:32,248 INFO [download_live_ocean] running in process 755857
2024-07-24 09:16:32,249 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:32,250 INFO [download_live_ocean] publishing log messages to tcp://*:5565
2024-07-24 09:16:32,251 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:32,252 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-07-24
2024-07-24 09:16:32,266 INFO [collect_river_data] running in process 755851
2024-07-24 09:16:32,267 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:32,268 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-24 09:16:32,269 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:32,270 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-07-23
2024-07-24 09:16:32,280 INFO [collect_river_data] running in process 755852
2024-07-24 09:16:32,281 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:32,281 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-24 09:16:32,282 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:32,283 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-07-23
2024-07-24 09:16:32,285 INFO [collect_river_data] running in process 755853
2024-07-24 09:16:32,286 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:32,286 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-24 09:16:32,287 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:32,288 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-07-23
2024-07-24 09:16:32,344 INFO [make_turbidity_file] running in process 755854
2024-07-24 09:16:32,345 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:32,346 INFO [make_turbidity_file] publishing log messages to tcp://*:5570
2024-07-24 09:16:32,347 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:32,348 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-07-24
2024-07-24 09:16:32,416 INFO [collect_river_data] running in process 755849
2024-07-24 09:16:32,418 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:32,419 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-24 09:16:32,420 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:32,421 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-07-23
2024-07-24 09:16:32,499 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-07-23&endDT=2024-07-23 "HTTP/1.1 200 OK"
2024-07-24 09:16:32,501 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-07-23&endDT=2024-07-23 "HTTP/1.1 200 OK"
2024-07-24 09:16:32,506 CRITICAL [collect_river_data] NisquallyMcKenna 2024-07-23 timeSeries is empty
2024-07-24 09:16:32,956 CRITICAL [collect_river_data] SnohomishMonroe 2024-07-23 timeSeries is empty
2024-07-24 09:16:33,087 INFO [crop_gribs] running in process 755859
2024-07-24 09:16:33,088 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-07-23&endDT=2024-07-23 "HTTP/1.1 200 OK"
2024-07-24 09:16:33,089 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:33,091 INFO [crop_gribs] publishing log messages to tcp://*:5572
2024-07-24 09:16:33,093 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:33,095 INFO [crop_gribs] cropping 2024-07-24 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-07-24 09:16:33,096 CRITICAL [collect_river_data] GreenwaterGreenwater 2024-07-23 timeSeries is empty
2024-07-24 09:16:33,123 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240724/18/
2024-07-24 09:16:33,125 CRITICAL [collect_river_data] Calculation of USGS GreenwaterGreenwater river average discharge for 2024-07-23 or appending it to GreenwaterGreenwater_flow file failed
2024-07-24 09:16:33,155 CRITICAL [collect_river_data] Calculation of USGS NisquallyMcKenna river average discharge for 2024-07-23 or appending it to NisquallyMcKenna_flow file failed
2024-07-24 09:16:33,178 CRITICAL [collect_river_data] Calculation of USGS SnohomishMonroe river average discharge for 2024-07-23 or appending it to SnohomishMonroe_flow file failed
2024-07-24 09:16:33,442 INFO [collect_NeahBay_ssh] 2024-07-24T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-07-24 09:16:33,445 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-24 09:16:33,454 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 09:16:33,741 INFO [download_live_ocean] downloaded 10109557 bytes from https://liveocean.apl.uw.edu/output/f2024.07.24/ubc.nc to /results/forcing/LiveOcean/downloaded/20240724/low_passed_UBC.nc
2024-07-24 09:16:33,742 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-07-24 09:16:34,218 INFO [grib_to_netcdf] running in process 755989
2024-07-24 09:16:34,221 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:34,222 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2024-07-24 09:16:34,223 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:34,224 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-24 nowcast and forecast runs
2024-07-24 09:16:34,767 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240724/low_passed_UBC.nc
2024-07-24 09:16:34,768 INFO [download_live_ocean] 2024-07-24 Live Ocean file for Salish Sea western boundary downloaded
2024-07-24 09:16:34,770 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-07-24 09:16:34,927 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-07-24'], host='localhost')
2024-07-24 09:16:35,389 WARNING [make_turbidity_file] Insufficient data to proceed to turbidity interpolation; cannot create Fraser River turbidity file for 2024-07-24
2024-07-24 09:16:35,403 INFO [make_turbidity_file] 2024-07-24 Fraser River turbidity file creation complete
2024-07-24 09:16:36,047 INFO [make_ssh_files] running in process 756036
2024-07-24 09:16:36,048 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:36,049 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2024-07-24 09:16:36,050 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:36,051 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-07-24 NOAA Neah Bay 06Z observation and forecast values
2024-07-24 09:16:37,045 INFO [make_live_ocean_files] running in process 756060
2024-07-24 09:16:37,046 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:37,047 INFO [make_live_ocean_files] publishing log messages to tcp://*:5567
2024-07-24 09:16:37,048 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:37,061 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-07-24 Live Ocean run
2024-07-24 09:16:38,583 CRITICAL [collect_river_data] Error while requesting https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-07-23&endDT=2024-07-23
2024-07-24 09:16:38,971 CRITICAL [collect_river_data] Calculation of USGS SkagitMountVernon river average discharge for 2024-07-23 or appending it to SkagitMountVernon_flow file failed
2024-07-24 09:16:40,426 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-24 09:16:40,609 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-24 09:16:40,790 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d22.nc
2024-07-24 09:16:40,971 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d23.nc
2024-07-24 09:16:41,172 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d24.nc
2024-07-24 09:16:41,355 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d25.nc
2024-07-24 09:16:41,535 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d26.nc
2024-07-24 09:16:41,718 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d27.nc
2024-07-24 09:16:41,829 INFO [make_ssh_files] sea surface height boundary file for 2024-07-24 nowcast run created
2024-07-24 09:16:41,833 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-24 09:16:41,843 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-24 09:16:41,847 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-24 09:16:42,572 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m07d24.nc
2024-07-24 09:16:43,128 INFO [make_runoff_file] running in process 756110
2024-07-24 09:16:43,129 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:43,130 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2024-07-24 09:16:43,131 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:43,169 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-24 09:16:43,233 INFO [make_runoff_file] 2024-07-24 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-24 09:16:43,235 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-24 09:16:43,628 INFO [make_v202111_runoff_file] running in process 756108
2024-07-24 09:16:43,630 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:16:43,631 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5565
2024-07-24 09:16:43,632 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:16:43,633 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-23
2024-07-24 09:16:44,502 ERROR [make_v202111_runoff_file] no 2024-07-23 discharge obs for Skagit_MountVernon: patching
2024-07-24 09:16:44,970 ERROR [make_v202111_runoff_file] no 2024-07-23 discharge obs for Snohomish_Monroe: patching
2024-07-24 09:16:45,165 ERROR [make_v202111_runoff_file] no 2024-07-23 discharge obs for Nisqually_McKenna: patching
2024-07-24 09:16:45,254 ERROR [make_v202111_runoff_file] no 2024-07-23 discharge obs for Greenwater_Greenwater: patching
2024-07-24 09:16:46,564 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-23: /results/forcing/rivers/R202108Dailies_y2024m07d23.nc
2024-07-24 09:16:46,565 INFO [make_v202111_runoff_file] 2024-07-23 runoff file creation completed
2024-07-24 09:16:46,567 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-24 09:16:49,519 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d25.nc
2024-07-24 09:16:53,140 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d26.nc
2024-07-24 09:16:53,151 INFO [grib_to_netcdf] 2024-07-24 NEMO-atmos forcing file for nowcast+ created
2024-07-24 09:16:53,155 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-24 09:17:17,082 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d24.nc
2024-07-24 09:17:17,083 INFO [make_live_ocean_files] 2024-07-24 Live Ocean western boundary conditions files created
2024-07-24 09:17:17,085 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-07-24 09:17:17,091 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-07-24 09:17:17,096 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-07-24 09:17:17,100 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-07-24 09:17:17,105 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-07-24 09:17:17,109 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-07-24 09:17:18,195 INFO [ping_erddap] running in process 756152
2024-07-24 09:17:18,196 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:17:18,197 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-24 09:17:18,198 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:17:18,199 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-07-24 09:17:18,202 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-24 09:17:18,267 INFO [upload_forcing] running in process 756157
2024-07-24 09:17:18,268 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:17:18,269 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-07-24 09:17:18,271 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:17:18,296 INFO [upload_forcing] running in process 756154
2024-07-24 09:17:18,297 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:17:18,299 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-24 09:17:18,300 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:17:18,322 INFO [upload_forcing] running in process 756156
2024-07-24 09:17:18,323 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:17:18,324 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-24 09:17:18,325 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:17:18,410 INFO [upload_forcing] running in process 756155
2024-07-24 09:17:18,412 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:17:18,413 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-24 09:17:18,414 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:17:20,291 INFO [upload_forcing] nowcast+ 2024-07-24 forcing files upload to optimum-hindcast completed
2024-07-24 09:17:20,296 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 09:17:20,813 INFO [upload_forcing] nowcast+ 2024-07-24 forcing files upload to orcinus-nowcast-agrif completed
2024-07-24 09:17:20,816 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 09:17:28,582 INFO [upload_forcing] nowcast+ 2024-07-24 forcing files upload to robot.graham completed
2024-07-24 09:17:28,587 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 09:17:33,456 INFO [upload_forcing] nowcast+ 2024-07-24 forcing files upload to arbutus.cloud-nowcast completed
2024-07-24 09:17:33,460 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 09:17:33,473 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 09:17:34,911 INFO [make_forcing_links] running in process 756192
2024-07-24 09:17:34,913 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:17:34,914 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-07-24 09:17:34,915 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:17:36,109 INFO [make_forcing_links] nowcast+ 2024-07-24 forcing file links on arbutus.cloud-nowcast created
2024-07-24 09:17:36,114 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-24 09:17:36,127 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 09:17:38,426 INFO [run_NEMO] running in process 16708
2024-07-24 09:17:38,427 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:17:38,429 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-24 09:17:38,545 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:17:39,339 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/24jul24nowcast_2024-07-24T091738.483088-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-24 09:17:39,354 INFO [run_NEMO] nowcast NEMO run for 2024-07-24 on arbutus.cloud-nowcast started
2024-07-24 09:17:39,357 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-24 09:17:39,373 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-24 09:17:41,472 INFO [watch_NEMO] running in process 16960
2024-07-24 09:17:41,473 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:17:41,474 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-24 09:17:41,480 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:17:41,712 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-24 09:22:41,705 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17805330 = 2024-07-24 04:59:20 UTC, 20.8% complete
2024-07-24 09:27:41,701 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17805794 = 2024-07-24 10:08:40 UTC, 42.3% complete
2024-07-24 09:32:41,673 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17806263 = 2024-07-24 15:21:20 UTC, 64.0% complete
2024-07-24 09:37:41,674 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17806769 = 2024-07-24 20:58:40 UTC, 87.4% complete
2024-07-24 09:42:41,681 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-07-24 09:42:41,684 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-24 09:42:41,916 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 09:42:41,925 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 09:42:43,335 INFO [download_results] running in process 756273
2024-07-24 09:42:43,336 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:42:43,337 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-24 09:42:43,338 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:42:43,339 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/24jul24 to /results/SalishSea/nowcast-blue.202111
2024-07-24 09:42:43,341 INFO [make_forcing_links] running in process 756272
2024-07-24 09:42:43,341 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:42:43,342 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-24 09:42:43,343 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:42:44,637 INFO [make_forcing_links] ssh 2024-07-24 forcing file links on arbutus.cloud-nowcast created
2024-07-24 09:42:44,641 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-24 09:42:44,653 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 09:42:46,831 INFO [run_NEMO] running in process 17346
2024-07-24 09:42:46,834 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:42:46,835 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-24 09:42:46,836 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:42:47,757 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/24jul24forecast_2024-07-24T094246.894274-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-24 09:42:47,778 INFO [run_NEMO] forecast NEMO run for 2024-07-24 on arbutus.cloud-nowcast started
2024-07-24 09:42:47,780 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-24 09:42:47,791 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-24 09:42:50,027 INFO [watch_NEMO] running in process 17598
2024-07-24 09:42:50,028 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:42:50,030 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-24 09:42:50,031 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:42:50,112 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-24 09:47:50,451 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17807472 = 2024-07-25 04:47:20 UTC, 13.3% complete
2024-07-24 09:52:50,146 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17807965 = 2024-07-25 10:16:00 UTC, 28.5% complete
2024-07-24 09:55:53,384 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 09:55:53,554 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-26'], host='localhost')
2024-07-24 09:55:53,563 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 09:55:53,569 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 09:55:55,027 INFO [download_results] running in process 756965
2024-07-24 09:55:55,029 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:55:55,030 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-24 09:55:55,032 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:55:55,033 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 09:55:59,240 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 09:56:56,274 INFO [download_results] hindcast 2018-11-26 results files from optimum-hindcast downloaded
2024-07-24 09:56:56,286 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 09:56:56,302 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-26'], host='localhost')
2024-07-24 09:56:57,818 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 09:56:57,834 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-nov', 'robot.graham'], host='localhost')
2024-07-24 09:56:59,041 INFO [archive_tarball] running in process 757011
2024-07-24 09:56:59,042 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 09:56:59,043 INFO [archive_tarball] publishing log messages to tcp://*:5565
2024-07-24 09:56:59,044 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-24 09:56:59,045 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-nov18.tar from /ocean/sallen/timbucktwo/oxygen/*nov18/
2024-07-24 09:57:50,354 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17808449 = 2024-07-25 15:38:40 UTC, 43.5% complete
2024-07-24 10:02:50,102 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17808914 = 2024-07-25 20:48:40 UTC, 57.8% complete
2024-07-24 10:03:21,715 INFO [download_results] nowcast 2024-07-24 results files from arbutus.cloud-nowcast downloaded
2024-07-24 10:03:21,721 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 10:03:21,735 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:03:21,740 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-07-23'], host='localhost')
2024-07-24 10:03:21,745 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:03:23,829 INFO [make_CHS_currents_file] running in process 757065
2024-07-24 10:03:23,830 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:03:23,831 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-07-24 10:03:23,832 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:03:25,151 INFO [make_plots] running in process 757064
2024-07-24 10:03:25,153 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:03:25,154 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-24 10:03:25,155 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:03:25,232 INFO [make_plots] running in process 757062
2024-07-24 10:03:25,233 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:03:25,234 INFO [make_plots] publishing log messages to tcp://*:5568
2024-07-24 10:03:25,235 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:03:51,721 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-24 for nowcast
2024-07-24 10:03:51,724 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-24 10:07:50,134 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17809362 = 2024-07-26 01:47:20 UTC, 71.7% complete
2024-07-24 10:09:04,124 INFO [make_plots] preparing render list for 2024-07-23 NEMO nowcast-blue comparison figures
2024-07-24 10:09:04,711 INFO [make_plots] preparing render list for 2024-07-24 NEMO nowcast-blue research figures
2024-07-24 10:09:05,458 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-blue research render list
2024-07-24 10:09:06,056 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-blue research render list
2024-07-24 10:09:06,057 INFO [make_plots] added Currents_sections_and_surface figure to 2024-07-24 NEMO nowcast-blue research render list
2024-07-24 10:09:06,058 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-07-24 NEMO nowcast-blue research render list
2024-07-24 10:09:06,059 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-07-24 NEMO nowcast-blue research render list
2024-07-24 10:09:06,061 INFO [make_plots] starting to render nemo nowcast research 24jul24 figures
2024-07-24 10:10:11,377 INFO [make_plots] finished rendering nemo nowcast research 24jul24 figures
2024-07-24 10:10:11,380 INFO [make_plots] nemo research plots for 2024-07-24 nowcast completed
2024-07-24 10:10:11,400 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-24 10:12:50,054 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17809815 = 2024-07-26 06:49:20 UTC, 85.6% complete
2024-07-24 10:17:16,843 INFO [make_plots] added SH_wind figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,845 INFO [make_plots] added Compare_VENUS_East figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,846 INFO [make_plots] added Compare_VENUS_Central figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,847 INFO [make_plots] added Central_ADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,849 INFO [make_plots] added Central_depavADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,850 INFO [make_plots] added Central_timeavADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,851 INFO [make_plots] added East_ADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,852 INFO [make_plots] added East_depavADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,854 INFO [make_plots] added East_timeavADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,855 INFO [make_plots] added ddl_ADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,857 INFO [make_plots] added ddl_depavADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,858 INFO [make_plots] added ddl_timeavADCP figure to 2024-07-23 NEMO nowcast-blue comparison render list
2024-07-24 10:17:16,860 INFO [make_plots] starting to render nemo nowcast comparison 23jul24 figures
2024-07-24 10:17:50,050 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17810280 = 2024-07-26 11:59:20 UTC, 100.0% complete
2024-07-24 10:18:36,114 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available
2024-07-24 10:18:36,665 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-24 10:18:37,172 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-24 10:18:37,670 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-24 10:18:38,172 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-24 10:18:38,674 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-24 10:18:39,189 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-24 10:18:39,713 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-24 10:18:40,220 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-24 10:18:40,722 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-24 10:18:40,724 INFO [make_plots] finished rendering nemo nowcast comparison 23jul24 figures
2024-07-24 10:18:40,807 INFO [make_plots] nemo comparison plots for 2024-07-23 nowcast completed
2024-07-24 10:18:40,811 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-24 10:22:41,421 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-nov18.index from /ocean/dlatorne/oxygen-nov18.tar
2024-07-24 10:22:41,504 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-nov18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-24 10:22:50,048 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-24 10:22:50,049 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-07-24 10:22:50,207 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 10:22:50,212 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 10:22:50,216 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:22:50,221 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:22:50,226 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:22:50,230 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:22:50,234 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:22:51,911 INFO [upload_forcing] running in process 757252
2024-07-24 10:22:51,913 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:51,913 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-07-24 10:22:51,914 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:51,918 INFO [download_results] running in process 757253
2024-07-24 10:22:51,920 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:51,920 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-24 10:22:51,921 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:51,922 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/24jul24 to /results/SalishSea/forecast.202111
2024-07-24 10:22:51,923 INFO [upload_forcing] running in process 757249
2024-07-24 10:22:51,924 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:51,925 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-24 10:22:51,926 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:51,929 INFO [upload_forcing] running in process 757250
2024-07-24 10:22:51,930 INFO [upload_forcing] running in process 757251
2024-07-24 10:22:51,931 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:51,932 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-24 10:22:51,932 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:51,933 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:51,934 INFO [upload_forcing] publishing log messages to tcp://*:5570
2024-07-24 10:22:51,935 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:52,585 CRITICAL [upload_forcing] Fraser River turbidity forcing file not found; created symlink to /results/forcing/rivers/river_turb/riverTurbDaily2_y2024m07d23.nc
2024-07-24 10:22:52,874 INFO [upload_forcing] turbidity 2024-07-24 forcing files upload to orcinus-nowcast-agrif completed
2024-07-24 10:22:52,879 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 10:22:52,897 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:22:53,032 INFO [upload_forcing] turbidity 2024-07-24 forcing files upload to optimum-hindcast completed
2024-07-24 10:22:53,035 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 10:22:53,141 INFO [upload_forcing] turbidity 2024-07-24 forcing files upload to arbutus.cloud-nowcast completed
2024-07-24 10:22:53,146 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 10:22:53,162 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:22:53,232 INFO [make_ww3_wind_file] running in process 18090
2024-07-24 10:22:53,233 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:53,234 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-24 10:22:53,303 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:53,304 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-24 forecast run
2024-07-24 10:22:53,307 INFO [make_ww3_current_file] running in process 18096
2024-07-24 10:22:53,308 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:53,309 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-24 10:22:53,310 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:53,311 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-24 forecast run
2024-07-24 10:22:54,163 INFO [make_forcing_links] running in process 757288
2024-07-24 10:22:54,165 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:54,166 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-07-24 10:22:54,167 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:54,553 INFO [make_forcing_links] running in process 757291
2024-07-24 10:22:54,554 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:54,556 INFO [make_forcing_links] publishing log messages to tcp://*:5568
2024-07-24 10:22:54,557 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:22:55,068 INFO [make_forcing_links] nowcast-agrif 2024-07-24 forcing file links on orcinus-nowcast-agrif created
2024-07-24 10:22:55,072 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-24 10:22:55,092 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:22:55,750 INFO [upload_forcing] turbidity 2024-07-24 forcing files upload to robot.graham completed
2024-07-24 10:22:55,755 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 10:22:56,308 INFO [run_NEMO_agrif] running in process 757306
2024-07-24 10:22:56,310 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:22:56,311 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5562
2024-07-24 10:22:56,312 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:23:00,143 INFO [make_forcing_links] nowcast-green 2024-07-24 forcing file links on arbutus.cloud-nowcast created
2024-07-24 10:23:00,148 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-24 10:23:00,166 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 10:23:04,116 INFO [run_NEMO] running in process 18866
2024-07-24 10:23:04,117 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:23:04,118 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-24 10:23:04,119 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:23:04,152 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-24 10:23:04,208 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-24 forecast run
2024-07-24 10:23:05,194 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/24jul24nowcast-green_2024-07-24T102304.209428-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-24 10:23:05,224 INFO [run_NEMO] nowcast-green NEMO run for 2024-07-24 on arbutus.cloud-nowcast started
2024-07-24 10:23:05,229 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-24 10:23:05,249 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-07-24 10:23:07,872 INFO [run_NEMO_agrif] job id for 24jul24nowcast-agrif: 11101987.orca2.ibb
2024-07-24 10:23:07,873 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-07-24 queued on orcinus-nowcast-agrif
2024-07-24 10:23:07,878 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-07-24 10:23:07,901 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11101987.orca2.ibb'], host='localhost')
2024-07-24 10:23:08,014 INFO [watch_NEMO] running in process 19121
2024-07-24 10:23:08,015 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:23:08,016 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-24 10:23:08,018 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:23:08,212 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-24 10:23:09,105 INFO [watch_NEMO_agrif] running in process 757324
2024-07-24 10:23:09,106 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:23:09,107 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5562
2024-07-24 10:23:09,108 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:23:09,914 INFO [watch_NEMO_agrif] watching 24jul24nowcast-agrif job 11101987 on orcinus-nowcast-agrif
2024-07-24 10:23:10,146 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif job 11101987 is queued on orcinus-nowcast-agrif
2024-07-24 10:23:52,388 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-24 forecast run
2024-07-24 10:23:52,394 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-24 10:23:52,416 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 10:23:56,087 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-24 10:23:56,110 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-24 10:23:59,755 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-24 10:24:11,050 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-07-24 10:28:08,256 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17805033 = 2024-07-24 01:41:20 UTC, 7.0% complete
2024-07-24 10:28:59,737 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 81, 18.8% complete
2024-07-24 10:29:11,521 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7825859 = 2024-07-24 01:58:40 UTC, 8.2% complete
2024-07-24 10:33:08,269 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17805203 = 2024-07-24 03:34:40 UTC, 14.9% complete
2024-07-24 10:33:59,879 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 165, 38.2% complete
2024-07-24 10:34:11,961 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7826064 = 2024-07-24 04:15:20 UTC, 17.7% complete
2024-07-24 10:38:08,189 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17805374 = 2024-07-24 05:28:40 UTC, 22.8% complete
2024-07-24 10:38:59,758 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 261, 60.4% complete
2024-07-24 10:39:12,425 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7826272 = 2024-07-24 06:34:00 UTC, 27.4% complete
2024-07-24 10:43:08,109 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17805547 = 2024-07-24 07:24:00 UTC, 30.8% complete
2024-07-24 10:43:59,770 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 354, 81.9% complete
2024-07-24 10:44:12,897 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7826482 = 2024-07-24 08:54:00 UTC, 37.1% complete
2024-07-24 10:48:08,133 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17805719 = 2024-07-24 09:18:40 UTC, 38.8% complete
2024-07-24 10:48:59,665 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-24 10:49:13,371 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7826689 = 2024-07-24 11:12:00 UTC, 46.7% complete
2024-07-24 10:52:21,024 INFO [download_results] forecast 2024-07-24 results files from arbutus.cloud-nowcast downloaded
2024-07-24 10:52:21,029 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 10:52:21,047 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:52:23,114 INFO [make_CHS_currents_file] running in process 757433
2024-07-24 10:52:23,115 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:52:23,116 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5567
2024-07-24 10:52:23,118 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:53:05,868 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-24 for forecast
2024-07-24 10:53:05,872 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-24 10:53:05,892 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:53:06,979 INFO [update_forecast_datasets] running in process 757458
2024-07-24 10:53:06,981 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:53:06,982 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2024-07-24 10:53:06,983 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:53:06,983 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/24jul24 run
2024-07-24 10:53:07,247 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/24jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-24 10:53:07,248 INFO [update_forecast_datasets] nemo 2024-07-24 forecast rolling forecast datasets updated
2024-07-24 10:53:07,250 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-24 10:53:07,269 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-24 10:53:07,274 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:53:07,278 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:53:08,023 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17805891 = 2024-07-24 11:13:20 UTC, 46.8% complete
2024-07-24 10:53:08,523 INFO [ping_erddap] running in process 757463
2024-07-24 10:53:08,525 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:53:08,526 INFO [ping_erddap] publishing log messages to tcp://*:5567
2024-07-24 10:53:08,527 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:53:08,534 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-24 10:53:08,539 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-24 10:53:09,503 INFO [make_surface_current_tiles] running in process 757466
2024-07-24 10:53:09,504 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:53:09,505 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5567
2024-07-24 10:53:09,506 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:53:09,508 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-24 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/24jul24
2024-07-24 10:53:11,437 INFO [make_plots] running in process 757465
2024-07-24 10:53:11,438 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:53:11,439 INFO [make_plots] publishing log messages to tcp://*:5568
2024-07-24 10:53:11,441 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:53:11,999 INFO [make_plots] preparing render list for 2024-07-24 NEMO forecast publish figures
2024-07-24 10:53:12,118 INFO [make_plots] added Website_thumbnail figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,120 INFO [make_plots] added Threshold_website figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,121 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,122 INFO [make_plots] added Boundary Bay figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,123 INFO [make_plots] added Campbell River figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,124 INFO [make_plots] added Cherry Point figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,125 INFO [make_plots] added Friday Harbor figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,126 INFO [make_plots] added Halfmoon Bay figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,127 INFO [make_plots] added Nanaimo figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,127 INFO [make_plots] added Neah Bay figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,128 INFO [make_plots] added New Westminster figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,129 INFO [make_plots] added Patricia Bay figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,129 INFO [make_plots] added Point Atkinson figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,130 INFO [make_plots] added Port Renfrew figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,130 INFO [make_plots] added Sand Heads figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,131 INFO [make_plots] added Sandy Cove figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,131 INFO [make_plots] added Squamish figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,132 INFO [make_plots] added Victoria figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,133 INFO [make_plots] added Woodwards Landing figure to 2024-07-24 NEMO forecast publish render list
2024-07-24 10:53:12,133 INFO [make_plots] starting to render nemo forecast publish 24jul24 figures
2024-07-24 10:53:59,525 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-24 10:53:59,526 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-24 10:53:59,702 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 10:53:59,707 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 10:54:01,092 INFO [download_wwatch3_results] running in process 757709
2024-07-24 10:54:01,092 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:54:01,093 INFO [download_wwatch3_results] publishing log messages to tcp://*:5569
2024-07-24 10:54:01,094 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:54:01,795 INFO [run_ww3] running in process 19575
2024-07-24 10:54:01,796 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:54:01,797 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-07-24 10:54:01,798 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:54:01,802 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-07-24T105402.036000-0700
2024-07-24 10:54:01,803 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-07-24T105402.036000-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-07-24 10:54:01,823 INFO [run_ww3] forecast WaveWatch3 run for 2024-07-24 on arbutus.cloud-nowcast started
2024-07-24 10:54:01,827 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-24 10:54:01,858 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-24 10:54:03,908 INFO [watch_ww3] running in process 19730
2024-07-24 10:54:03,909 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 10:54:03,909 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-07-24 10:54:03,910 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-24 10:54:03,947 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-24 10:54:13,841 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7826898 = 2024-07-24 13:31:20 UTC, 56.4% complete
2024-07-24 10:56:01,632 INFO [download_wwatch3_results] nowcast 2024-07-24 results files from arbutus.cloud-nowcast downloaded
2024-07-24 10:56:01,634 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-24 10:58:08,003 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17806061 = 2024-07-24 13:06:40 UTC, 54.7% complete
2024-07-24 10:58:34,047 INFO [archive_tarball] hindcast *nov18 results files archived to robot.graham
2024-07-24 10:58:34,050 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-24 10:59:03,927 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 75, 11.6% complete
2024-07-24 10:59:14,317 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7827107 = 2024-07-24 15:50:40 UTC, 66.0% complete
2024-07-24 11:03:07,987 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17806231 = 2024-07-24 15:00:00 UTC, 62.5% complete
2024-07-24 11:03:44,330 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/24jul24
2024-07-24 11:04:03,910 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 165, 25.5% complete
2024-07-24 11:04:14,800 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7827314 = 2024-07-24 18:08:40 UTC, 75.6% complete
2024-07-24 11:06:27,408 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-24 11:06:27,409 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-24 11:07:02,875 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:08:08,113 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17806401 = 2024-07-24 16:53:20 UTC, 70.4% complete
2024-07-24 11:08:14,588 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/24jul24
2024-07-24 11:08:14,600 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-24 forecast into /results/nowcast-sys/figures/surface_currents/forecast/24jul24
2024-07-24 11:08:14,601 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-24 forecast completed
2024-07-24 11:08:14,885 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-24 11:08:49,794 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:08:50,146 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:09:04,171 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 264, 40.7% complete
2024-07-24 11:09:15,275 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7827523 = 2024-07-24 20:28:00 UTC, 85.3% complete
2024-07-24 11:09:25,308 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:10:34,887 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:11:09,945 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:11:45,722 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:12:20,705 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:12:55,491 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:13:08,246 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17806571 = 2024-07-24 18:46:40 UTC, 78.3% complete
2024-07-24 11:13:30,764 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:14:04,174 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 354, 54.6% complete
2024-07-24 11:14:06,190 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:14:06,544 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:14:15,749 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7827733 = 2024-07-24 22:48:00 UTC, 95.0% complete
2024-07-24 11:14:41,526 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:15:16,334 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-19 00:00:00Z to 2024-07-26 11:55:00Z
2024-07-24 11:15:51,572 INFO [make_plots] finished rendering nemo forecast publish 24jul24 figures
2024-07-24 11:15:51,574 INFO [make_plots] nemo publish plots for 2024-07-24 forecast completed
2024-07-24 11:15:51,583 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-24 11:15:51,749 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 11:15:54,208 INFO [make_feeds] running in process 759515
2024-07-24 11:15:54,209 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:15:54,210 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-07-24 11:15:54,211 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:16:22,474 INFO [make_feeds] ATOM feeds for 2024-07-24 forecast run completed
2024-07-24 11:16:22,477 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-24 11:18:08,292 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17806744 = 2024-07-24 20:42:00 UTC, 86.3% complete
2024-07-24 11:19:04,213 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 444, 68.5% complete
2024-07-24 11:19:16,173 INFO [watch_NEMO_agrif] 24jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7827840 = 2024-07-24 23:59:20 UTC, 100.0% complete
2024-07-24 11:21:03,576 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 11:21:03,894 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-01'], host='localhost')
2024-07-24 11:21:03,901 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 11:21:03,906 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 11:21:05,137 INFO [download_results] running in process 759550
2024-07-24 11:21:05,139 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:21:05,140 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-24 11:21:05,141 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:21:05,142 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 11:21:09,219 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 11:21:44,791 INFO [download_results] hindcast 2018-12-01 results files from optimum-hindcast downloaded
2024-07-24 11:21:44,801 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 11:21:44,974 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-01'], host='localhost')
2024-07-24 11:21:46,361 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 11:23:08,294 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17806914 = 2024-07-24 22:35:20 UTC, 94.2% complete
2024-07-24 11:24:04,222 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 543, 83.8% complete
2024-07-24 11:24:16,390 INFO [watch_NEMO_agrif] NEMO AGRIF run 11101987.orca2.ibb on orcinus-nowcast-agrif completed
2024-07-24 11:24:16,395 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-07-24 11:24:16,722 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 11:24:18,259 INFO [download_results] running in process 759601
2024-07-24 11:24:18,260 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:24:18,262 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-24 11:24:18,263 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:24:18,264 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/24jul24 to /results/SalishSea/nowcast-agrif.201702
2024-07-24 11:24:41,652 INFO [download_results] nowcast-agrif 2024-07-24 results files from orcinus-nowcast-agrif downloaded
2024-07-24 11:24:41,659 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 11:24:41,829 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 11:24:45,352 INFO [make_plots] running in process 759608
2024-07-24 11:24:45,354 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:24:45,355 INFO [make_plots] publishing log messages to tcp://*:5562
2024-07-24 11:24:45,356 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:24:46,140 INFO [make_plots] preparing render list for 2024-07-24 NEMO nowcast-agrif research figures
2024-07-24 11:24:46,141 INFO [make_plots] added baynes_sound_surface figure to 2024-07-24 NEMO nowcast-agrif research render list
2024-07-24 11:24:46,142 INFO [make_plots] starting to render nemo nowcast-agrif research 24jul24 figures
2024-07-24 11:24:56,065 INFO [make_plots] finished rendering nemo nowcast-agrif research 24jul24 figures
2024-07-24 11:24:56,068 INFO [make_plots] nemo research plots for 2024-07-24 nowcast-agrif completed
2024-07-24 11:24:56,072 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-24 11:28:08,303 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17807040 = 2024-07-24 23:59:20 UTC, 100.0% complete
2024-07-24 11:29:04,226 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 639, 98.6% complete
2024-07-24 11:33:08,309 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-07-24 11:33:08,313 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-24 11:33:08,659 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 11:33:10,197 INFO [download_results] running in process 759718
2024-07-24 11:33:10,199 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:33:10,200 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-24 11:33:10,201 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:33:10,203 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/24jul24 to /results2/SalishSea/nowcast-green.202111
2024-07-24 11:34:04,868 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-24 11:34:04,871 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-24 11:34:05,198 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 11:34:06,337 INFO [download_wwatch3_results] running in process 759727
2024-07-24 11:34:06,338 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:34:06,340 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-07-24 11:34:06,341 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:42:59,928 INFO [download_wwatch3_results] forecast 2024-07-24 results files from arbutus.cloud-nowcast downloaded
2024-07-24 11:42:59,932 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-24 11:43:00,107 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 11:43:01,448 INFO [update_forecast_datasets] running in process 759749
2024-07-24 11:43:01,449 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:43:01,451 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566
2024-07-24 11:43:01,452 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:43:01,453 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/24jul24 run
2024-07-24 11:43:01,455 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/24jul24 run
2024-07-24 11:43:01,505 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/24jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-24 11:43:01,507 INFO [update_forecast_datasets] wwatch3 2024-07-24 forecast rolling forecast datasets updated
2024-07-24 11:43:01,509 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-24 11:43:01,687 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-24 11:43:02,849 INFO [ping_erddap] running in process 759754
2024-07-24 11:43:02,850 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:43:02,852 INFO [ping_erddap] publishing log messages to tcp://*:5566
2024-07-24 11:43:02,853 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:43:02,854 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-24 11:43:02,856 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-24 11:43:03,029 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 11:43:06,522 INFO [make_plots] running in process 759759
2024-07-24 11:43:06,524 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 11:43:06,525 INFO [make_plots] publishing log messages to tcp://*:5566
2024-07-24 11:43:06,526 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-24 11:43:06,527 INFO [make_plots] preparing render list for 2024-07-24 WaveWatch3 forecast publish figures
2024-07-24 11:43:06,528 INFO [make_plots] added Halibut Bank figure to 2024-07-24 WaveWatch3 forecast publish render list
2024-07-24 11:43:06,529 INFO [make_plots] added Sentry Shoal figure to 2024-07-24 WaveWatch3 forecast publish render list
2024-07-24 11:43:06,530 INFO [make_plots] starting to render wwatch3 forecast publish 24jul24 figures
2024-07-24 11:43:06,837 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-24 11:43:14,878 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-24 11:43:21,228 INFO [make_plots] finished rendering wwatch3 forecast publish 24jul24 figures
2024-07-24 11:43:21,229 INFO [make_plots] wwatch3 publish plots for 2024-07-24 forecast completed
2024-07-24 11:43:21,233 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-24 12:10:12,966 INFO [download_results] nowcast-green 2024-07-24 results files from arbutus.cloud-nowcast downloaded
2024-07-24 12:10:12,972 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 12:10:13,145 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 12:10:13,150 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-07-24 12:10:13,153 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 12:10:13,159 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 12:10:13,164 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 12:10:14,523 INFO [ping_erddap] running in process 759858
2024-07-24 12:10:14,524 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 12:10:14,525 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-24 12:10:14,526 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-24 12:10:14,531 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-07-24 12:10:14,532 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-24 12:10:15,586 INFO [make_averaged_dataset] running in process 759860
2024-07-24 12:10:15,587 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 12:10:15,588 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2024-07-24 12:10:15,589 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-24 12:10:15,590 INFO [make_averaged_dataset] creating day-averaged dataset for 24-Jul-2024 chemistry
2024-07-24 12:10:15,591 INFO [make_averaged_dataset] running in process 759861
2024-07-24 12:10:15,592 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 12:10:15,593 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2024-07-24 12:10:15,594 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-24 12:10:15,595 INFO [make_averaged_dataset] creating day-averaged dataset for 24-Jul-2024 physics
2024-07-24 12:10:15,598 INFO [make_averaged_dataset] running in process 759859
2024-07-24 12:10:15,599 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 12:10:15,600 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567
2024-07-24 12:10:15,601 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-24 12:10:15,602 INFO [make_averaged_dataset] creating day-averaged dataset for 24-Jul-2024 biology
2024-07-24 12:10:16,913 INFO [make_plots] running in process 759856
2024-07-24 12:10:16,914 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 12:10:16,915 INFO [make_plots] publishing log messages to tcp://*:5568
2024-07-24 12:10:16,916 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-24 12:10:17,590 INFO [make_plots] preparing render list for 2024-07-24 NEMO nowcast-green research figures
2024-07-24 12:10:19,311 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:20,047 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:20,775 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:21,490 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:22,219 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:22,930 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:23,629 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:24,473 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:25,281 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:26,077 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:26,881 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:27,653 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:28,337 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:28,338 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:28,339 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:28,340 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-07-24 NEMO nowcast-green research render list
2024-07-24 12:10:28,340 INFO [make_plots] starting to render nemo nowcast-green research 24jul24 figures
2024-07-24 12:11:08,250 INFO [make_averaged_dataset] day-averaged dataset for 24-Jul-2024 physics created
2024-07-24 12:11:08,253 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-24 12:11:37,674 INFO [make_averaged_dataset] day-averaged dataset for 24-Jul-2024 chemistry created
2024-07-24 12:11:37,676 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-24 12:11:51,500 INFO [make_averaged_dataset] day-averaged dataset for 24-Jul-2024 biology created
2024-07-24 12:11:51,503 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-24 12:51:13,977 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 12:51:14,324 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-06'], host='localhost')
2024-07-24 12:51:14,331 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 12:51:14,336 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 12:51:15,576 INFO [download_results] running in process 760259
2024-07-24 12:51:15,577 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 12:51:15,578 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-24 12:51:15,579 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 12:51:15,580 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 12:51:19,799 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 12:51:55,929 INFO [download_results] hindcast 2018-12-06 results files from optimum-hindcast downloaded
2024-07-24 12:51:55,939 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 12:51:56,114 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-06'], host='localhost')
2024-07-24 12:51:57,311 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 13:21:05,864 INFO [make_plots] finished rendering nemo nowcast-green research 24jul24 figures
2024-07-24 13:21:06,033 INFO [make_plots] nemo research plots for 2024-07-24 nowcast-green completed
2024-07-24 13:21:06,103 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-24 14:16:24,468 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 14:16:24,827 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-11'], host='localhost')
2024-07-24 14:16:24,834 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 14:16:24,839 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 14:16:26,333 INFO [download_results] running in process 760633
2024-07-24 14:16:26,335 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 14:16:26,336 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-24 14:16:26,337 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 14:16:26,338 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 14:16:30,576 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 14:17:06,265 INFO [download_results] hindcast 2018-12-11 results files from optimum-hindcast downloaded
2024-07-24 14:17:06,276 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 14:17:06,437 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-11'], host='localhost')
2024-07-24 14:17:07,766 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 15:09:15,457 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240724/18/
2024-07-24 15:09:15,458 INFO [collect_weather] 2024-07-24 2.5km weather forecast 18 collection complete
2024-07-24 15:09:15,460 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-24 15:09:15,817 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-07-24 15:09:15,823 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-07-24 15:09:15,829 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-07-24 15:09:15,833 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-07-25'], host='localhost')
2024-07-24 15:09:16,360 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240724/18/
2024-07-24 15:09:16,361 INFO [crop_gribs] 2024-07-24 18 GRIBs cropping complete
2024-07-24 15:09:16,364 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-24 15:09:17,173 INFO [download_weather] running in process 760806
2024-07-24 15:09:17,174 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 15:09:17,175 INFO [download_weather] publishing log messages to tcp://*:5562
2024-07-24 15:09:17,176 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-24 15:09:17,177 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240724
2024-07-24 15:09:17,178 INFO [collect_weather] running in process 760807
2024-07-24 15:09:17,178 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 15:09:17,179 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-07-24 15:09:17,180 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-24 15:09:17,181 INFO [download_weather] running in process 760804
2024-07-24 15:09:17,182 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 15:09:17,182 INFO [download_weather] publishing log messages to tcp://*:5565
2024-07-24 15:09:17,183 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-24 15:09:17,184 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240724
2024-07-24 15:09:17,201 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-07-24 15:09:18,622 INFO [crop_gribs] running in process 760808
2024-07-24 15:09:18,623 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 15:09:18,624 INFO [crop_gribs] publishing log messages to tcp://*:5567
2024-07-24 15:09:18,625 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-24 15:09:18,627 INFO [crop_gribs] cropping 2024-07-25 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-07-24 15:09:18,639 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240725/00/
2024-07-24 15:14:27,892 INFO [download_weather] 2024-07-24 1km weather forecast 00 downloads complete
2024-07-24 15:14:27,895 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-24 15:14:29,001 INFO [download_weather] 2024-07-24 1km weather forecast 12 downloads complete
2024-07-24 15:14:29,004 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-24 15:41:34,871 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 15:41:35,221 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-16'], host='localhost')
2024-07-24 15:41:35,228 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 15:41:35,231 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 15:41:36,387 INFO [download_results] running in process 761010
2024-07-24 15:41:36,388 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 15:41:36,389 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-24 15:41:36,390 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 15:41:36,391 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 15:41:40,634 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 15:42:16,362 INFO [download_results] hindcast 2018-12-16 results files from optimum-hindcast downloaded
2024-07-24 15:42:16,372 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 15:42:16,597 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-16'], host='localhost')
2024-07-24 15:42:17,782 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 17:06:44,694 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 17:06:45,062 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-21'], host='localhost')
2024-07-24 17:06:45,070 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 17:06:45,075 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 17:06:46,369 INFO [download_results] running in process 761706
2024-07-24 17:06:46,370 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 17:06:46,371 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-24 17:06:46,372 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 17:06:46,374 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 17:06:50,461 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 17:07:26,542 INFO [download_results] hindcast 2018-12-21 results files from optimum-hindcast downloaded
2024-07-24 17:07:26,553 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 17:07:26,764 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-21'], host='localhost')
2024-07-24 17:07:28,114 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 18:51:56,812 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 18:51:57,164 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-26'], host='localhost')
2024-07-24 18:51:57,171 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 18:51:57,176 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 18:51:58,388 INFO [download_results] running in process 762441
2024-07-24 18:51:58,389 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 18:51:58,390 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-24 18:51:58,391 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 18:51:58,392 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 18:52:02,629 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 18:52:43,526 INFO [download_results] hindcast 2018-12-26 results files from optimum-hindcast downloaded
2024-07-24 18:52:43,539 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 18:52:43,740 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-26'], host='localhost')
2024-07-24 18:52:45,019 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 18:52:45,221 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-dec', 'robot.graham'], host='localhost')
2024-07-24 18:52:46,321 INFO [archive_tarball] running in process 762530
2024-07-24 18:52:46,322 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 18:52:46,324 INFO [archive_tarball] publishing log messages to tcp://*:5565
2024-07-24 18:52:46,325 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-24 18:52:46,326 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-dec18.tar from /ocean/sallen/timbucktwo/oxygen/*dec18/
2024-07-24 18:58:21,961 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-dec18.index from /ocean/dlatorne/oxygen-dec18.tar
2024-07-24 18:58:22,047 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-dec18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-24 19:40:05,604 INFO [archive_tarball] hindcast *dec18 results files archived to robot.graham
2024-07-24 19:40:05,607 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-24 20:22:07,094 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 20:22:07,442 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-01-01'], host='localhost')
2024-07-24 20:22:07,450 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 20:22:07,454 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 20:22:08,857 INFO [download_results] running in process 763024
2024-07-24 20:22:08,859 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 20:22:08,860 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-24 20:22:08,861 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 20:22:08,862 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jan19 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 20:22:12,987 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 20:22:48,646 INFO [download_results] hindcast 2019-01-01 results files from optimum-hindcast downloaded
2024-07-24 20:22:48,655 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 20:22:48,857 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-01-01'], host='localhost')
2024-07-24 20:22:50,054 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 21:07:46,932 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240725/00/
2024-07-24 21:07:46,934 INFO [collect_weather] 2024-07-25 2.5km weather forecast 00 collection complete
2024-07-24 21:07:46,937 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-24 21:07:47,278 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-07-24 21:07:47,285 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-07-25'], host='localhost')
2024-07-24 21:07:48,494 INFO [collect_weather] running in process 763365
2024-07-24 21:07:48,495 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 21:07:48,496 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-07-24 21:07:48,497 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-24 21:07:48,512 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-07-24 21:07:48,612 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240725/00/
2024-07-24 21:07:48,613 INFO [crop_gribs] 2024-07-25 00 GRIBs cropping complete
2024-07-24 21:07:48,615 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-24 21:07:49,542 INFO [crop_gribs] running in process 763367
2024-07-24 21:07:49,543 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 21:07:49,545 INFO [crop_gribs] publishing log messages to tcp://*:5565
2024-07-24 21:07:49,546 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-24 21:07:49,547 INFO [crop_gribs] cropping 2024-07-25 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-07-24 21:07:49,561 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240725/06/
2024-07-24 21:47:17,462 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 21:47:17,797 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-01-06'], host='localhost')
2024-07-24 21:47:17,805 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 21:47:17,811 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 21:47:19,041 INFO [download_results] running in process 763478
2024-07-24 21:47:19,042 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 21:47:19,043 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-24 21:47:19,044 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 21:47:19,046 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06jan19 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 21:47:23,335 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 21:47:58,839 INFO [download_results] hindcast 2019-01-06 results files from optimum-hindcast downloaded
2024-07-24 21:47:58,850 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 21:47:59,050 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-01-06'], host='localhost')
2024-07-24 21:48:00,478 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 23:12:27,649 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 23:12:28,055 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-01-11'], host='localhost')
2024-07-24 23:12:28,063 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 23:12:28,069 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 23:12:29,299 INFO [download_results] running in process 763844
2024-07-24 23:12:29,300 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 23:12:29,302 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-24 23:12:29,303 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 23:12:29,304 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11jan19 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 23:12:33,644 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 23:13:08,933 INFO [download_results] hindcast 2019-01-11 results files from optimum-hindcast downloaded
2024-07-24 23:13:08,944 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 23:13:09,152 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-01-11'], host='localhost')
2024-07-24 23:13:10,540 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 00:42:38,096 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 00:42:38,445 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-01-16'], host='localhost')
2024-07-25 00:42:38,454 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 00:42:38,459 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 00:42:39,721 INFO [download_results] running in process 764276
2024-07-25 00:42:39,722 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 00:42:39,723 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-25 00:42:39,724 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 00:42:39,725 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16jan19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 00:42:43,948 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 00:43:19,539 INFO [download_results] hindcast 2019-01-16 results files from optimum-hindcast downloaded
2024-07-25 00:43:19,550 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 00:43:19,756 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-01-16'], host='localhost')
2024-07-25 00:43:21,113 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 02:07:48,524 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 02:07:48,874 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-01-21'], host='localhost')
2024-07-25 02:07:48,882 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 02:07:48,887 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 02:07:50,119 INFO [download_results] running in process 764590
2024-07-25 02:07:50,120 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 02:07:50,121 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-25 02:07:50,122 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 02:07:50,123 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21jan19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 02:07:54,272 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 02:08:29,826 INFO [download_results] hindcast 2019-01-21 results files from optimum-hindcast downloaded
2024-07-25 02:08:29,836 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 02:08:30,041 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-01-21'], host='localhost')
2024-07-25 02:08:31,319 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 03:02:53,224 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240725/06/
2024-07-25 03:02:53,225 INFO [collect_weather] 2024-07-25 2.5km weather forecast 06 collection complete
2024-07-25 03:02:53,228 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-25 03:02:53,586 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,592 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,598 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,603 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,607 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,611 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,616 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,622 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,626 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,631 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,642 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,648 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,651 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,662 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,668 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-07-25 03:02:53,673 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-07-25 03:02:53,686 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-07-25 03:02:53,693 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-07-24'], host='localhost')
2024-07-25 03:02:53,718 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-07-25 03:02:53,731 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-07-25 03:02:53,744 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-07-25 03:02:56,584 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240725/06/
2024-07-25 03:02:56,585 INFO [crop_gribs] 2024-07-25 06 GRIBs cropping complete
2024-07-25 03:02:56,588 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-25 03:02:56,667 INFO [collect_NeahBay_ssh] running in process 764938
2024-07-25 03:02:56,669 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:56,671 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-07-25 03:02:56,672 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:56,673 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-07-25
2024-07-25 03:02:56,698 INFO [collect_weather] running in process 764939
2024-07-25 03:02:56,700 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:56,701 INFO [collect_weather] publishing log messages to tcp://*:5566
2024-07-25 03:02:56,702 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:56,733 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-07-25 03:02:57,206 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-07-25 03:02:57,636 INFO [collect_river_data] running in process 764920
2024-07-25 03:02:57,639 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,639 INFO [collect_river_data] publishing log messages to tcp://*:5565
2024-07-25 03:02:57,640 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,641 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-07-24
2024-07-25 03:02:57,649 INFO [collect_river_data] running in process 764927
2024-07-25 03:02:57,650 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,651 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-25 03:02:57,658 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,659 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-07-24
2024-07-25 03:02:57,708 INFO [collect_river_data] running in process 764925
2024-07-25 03:02:57,709 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,710 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-25 03:02:57,711 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,712 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-07-24
2024-07-25 03:02:57,727 INFO [collect_river_data] running in process 764923
2024-07-25 03:02:57,728 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,729 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-25 03:02:57,730 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,731 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-07-24
2024-07-25 03:02:57,739 INFO [collect_river_data] running in process 764922
2024-07-25 03:02:57,740 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,742 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-25 03:02:57,745 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,746 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-07-24
2024-07-25 03:02:57,747 INFO [collect_river_data] running in process 764918
2024-07-25 03:02:57,748 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,749 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-25 03:02:57,749 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,750 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-07-24
2024-07-25 03:02:57,794 INFO [collect_river_data] running in process 764921
2024-07-25 03:02:57,795 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,796 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-07-25 03:02:57,798 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,798 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-07-24
2024-07-25 03:02:57,876 INFO [collect_river_data] running in process 764929
2024-07-25 03:02:57,876 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,877 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-07-25 03:02:57,878 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,879 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-07-24
2024-07-25 03:02:57,906 INFO [collect_river_data] running in process 764924
2024-07-25 03:02:57,907 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,909 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-07-25 03:02:57,910 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,911 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-07-24
2024-07-25 03:02:57,930 INFO [collect_river_data] running in process 764933
2024-07-25 03:02:57,932 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,934 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-07-25 03:02:57,935 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,936 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-07-24
2024-07-25 03:02:57,937 INFO [collect_river_data] running in process 764928
2024-07-25 03:02:57,939 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,941 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-07-25 03:02:57,942 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,944 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-07-24
2024-07-25 03:02:57,949 INFO [collect_river_data] running in process 764930
2024-07-25 03:02:57,950 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,951 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-07-25 03:02:57,952 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,953 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-07-24
2024-07-25 03:02:57,955 INFO [collect_river_data] running in process 764931
2024-07-25 03:02:57,956 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,956 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-07-25 03:02:57,957 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,958 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-07-24
2024-07-25 03:02:57,960 INFO [collect_river_data] running in process 764932
2024-07-25 03:02:57,961 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:57,962 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-07-25 03:02:57,962 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:57,963 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-07-24
2024-07-25 03:02:58,333 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-07-25 03:02:58,334 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-07-24 completed
2024-07-25 03:02:58,335 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-07-25 03:02:58,336 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-07-24 completed
2024-07-25 03:02:58,337 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:02:58,432 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-07-25 03:02:58,437 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-07-24 completed
2024-07-25 03:02:58,440 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-07-25 03:02:58,441 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-07-24 completed
2024-07-25 03:02:58,442 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-07-24 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-07-25 03:02:58,443 INFO [collect_river_data] ECCC Capilano river data collection for 2024-07-24 completed
2024-07-25 03:02:58,471 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-07-24 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-07-25 03:02:58,472 INFO [collect_river_data] ECCC Englishman river data collection for 2024-07-24 completed
2024-07-25 03:02:58,517 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-07-24 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-07-25 03:02:58,518 INFO [collect_river_data] ECCC Fraser river data collection for 2024-07-24 completed
2024-07-25 03:02:58,540 INFO [collect_NeahBay_ssh] 2024-07-25T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-07-25 03:02:58,542 INFO [get_vfpa_hadcp] running in process 764937
2024-07-25 03:02:58,543 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:58,544 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5590
2024-07-25 03:02:58,546 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:58,547 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-24
2024-07-25 03:02:58,589 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-07-25 03:02:58,590 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-07-24 completed
2024-07-25 03:02:58,597 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-07-25 03:02:58,598 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-07-24 completed
2024-07-25 03:02:58,625 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-07-25 03:02:58,625 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-07-24 completed
2024-07-25 03:02:58,645 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-07-25 03:02:58,646 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-07-24 completed
2024-07-25 03:02:58,677 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-07-25 03:02:58,678 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-07-24 completed
2024-07-25 03:02:58,698 INFO [crop_gribs] running in process 764940
2024-07-25 03:02:58,698 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:58,699 INFO [crop_gribs] publishing log messages to tcp://*:5591
2024-07-25 03:02:58,700 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:58,701 INFO [crop_gribs] cropping 2024-07-25 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-07-25 03:02:58,709 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-07-25 03:02:58,709 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-07-24 completed
2024-07-25 03:02:58,713 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240725/12/
2024-07-25 03:02:58,747 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-07-24 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-07-25 03:02:58,748 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-07-24 completed
2024-07-25 03:02:58,750 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:02:58,938 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:02:58,941 INFO [get_onc_ferry] running in process 764936
2024-07-25 03:02:58,941 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:58,942 INFO [get_onc_ferry] publishing log messages to tcp://*:5592
2024-07-25 03:02:58,943 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:58,943 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-07-24
2024-07-25 03:02:58,958 INFO [get_onc_ctd] running in process 764934
2024-07-25 03:02:58,959 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:58,959 INFO [get_onc_ctd] publishing log messages to tcp://*:5593
2024-07-25 03:02:58,960 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:58,961 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-07-24
2024-07-25 03:02:58,999 INFO [get_onc_ctd] running in process 764935
2024-07-25 03:02:59,000 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:59,001 INFO [get_onc_ctd] publishing log messages to tcp://*:5594
2024-07-25 03:02:59,002 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:59,003 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-07-24
2024-07-25 03:02:59,166 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:02:59,358 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:02:59,589 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:02:59,767 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:02:59,833 INFO [grib_to_netcdf] running in process 765298
2024-07-25 03:02:59,834 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:02:59,835 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565
2024-07-25 03:02:59,836 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:02:59,837 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-25 forecast2 run
2024-07-25 03:02:59,950 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-25 03:03:00,134 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 03:03:00,140 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:03:00,403 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:03:00,598 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:03:00,767 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-07-24
2024-07-25 03:03:00,794 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:03:00,862 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-24 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202407.nc
2024-07-25 03:03:00,863 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-07 netcdf file
2024-07-25 03:03:01,015 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:03:01,210 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:03:01,407 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-25 03:03:01,409 CRITICAL [get_onc_ctd] 2024-07-24 ONC SCVIP CTD T&S file creation failed
2024-07-25 03:03:01,667 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2024-07-25 03:03:01,789 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2024-07-25 03:03:02,927 INFO [ping_erddap] running in process 765419
2024-07-25 03:03:02,928 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:02,929 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-25 03:03:02,930 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:02,931 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2024-07-25 03:03:02,933 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 03:03:03,038 INFO [make_ssh_files] running in process 765382
2024-07-25 03:03:03,039 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:03,040 INFO [make_ssh_files] publishing log messages to tcp://*:5567
2024-07-25 03:03:03,041 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:03,043 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-07-25 NOAA Neah Bay 00Z observation and forecast values
2024-07-25 03:03:07,781 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-25 03:03:07,958 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-25 03:03:08,130 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d22.nc
2024-07-25 03:03:08,323 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d23.nc
2024-07-25 03:03:08,393 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d26.nc
2024-07-25 03:03:08,521 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d24.nc
2024-07-25 03:03:08,753 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d25.nc
2024-07-25 03:03:08,933 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d26.nc
2024-07-25 03:03:09,126 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d27.nc
2024-07-25 03:03:09,303 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d28.nc
2024-07-25 03:03:09,478 INFO [make_ssh_files] sea surface height boundary file for 2024-07-25 forecast2 run created
2024-07-25 03:03:09,481 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-25 03:03:09,665 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-25 03:03:09,670 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-25 03:03:10,737 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d27.nc
2024-07-25 03:03:10,743 INFO [grib_to_netcdf] 2024-07-25 NEMO-atmos forcing file for forecast2 created
2024-07-25 03:03:10,745 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-25 03:03:11,060 INFO [make_runoff_file] running in process 765452
2024-07-25 03:03:11,061 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:11,061 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-07-25 03:03:11,062 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:11,178 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-25 03:03:11,287 INFO [make_runoff_file] 2024-07-25 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-25 03:03:11,289 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-25 03:03:11,525 INFO [make_v202111_runoff_file] running in process 765450
2024-07-25 03:03:11,526 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:11,528 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5565
2024-07-25 03:03:11,528 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:11,529 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-24
2024-07-25 03:03:12,672 ERROR [make_v202111_runoff_file] no 2024-07-24 discharge obs for Skagit_MountVernon: patching
2024-07-25 03:03:13,150 ERROR [make_v202111_runoff_file] no 2024-07-24 discharge obs for Snohomish_Monroe: patching
2024-07-25 03:03:13,375 ERROR [make_v202111_runoff_file] no 2024-07-24 discharge obs for Nisqually_McKenna: patching
2024-07-25 03:03:13,482 ERROR [make_v202111_runoff_file] no 2024-07-24 discharge obs for Greenwater_Greenwater: patching
2024-07-25 03:03:14,891 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-24: /results/forcing/rivers/R202108Dailies_y2024m07d24.nc
2024-07-25 03:03:14,892 INFO [make_v202111_runoff_file] 2024-07-24 runoff file creation completed
2024-07-25 03:03:14,895 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-25 03:03:15,096 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-07-25 03:03:15,101 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-07-25 03:03:15,106 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-07-25 03:03:15,109 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-07-25 03:03:16,405 INFO [upload_forcing] running in process 765491
2024-07-25 03:03:16,407 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:16,408 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-07-25 03:03:16,409 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:16,440 INFO [upload_forcing] running in process 765492
2024-07-25 03:03:16,441 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:16,442 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-07-25 03:03:16,444 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:16,457 INFO [upload_forcing] running in process 765494
2024-07-25 03:03:16,459 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:16,460 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-25 03:03:16,461 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:16,492 INFO [upload_forcing] running in process 765493
2024-07-25 03:03:16,493 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:16,494 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-25 03:03:16,495 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:17,867 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d24.nc
2024-07-25 03:03:17,916 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-07-24
2024-07-25 03:03:17,939 INFO [upload_forcing] forecast2 2024-07-25 forcing files upload to optimum-hindcast completed
2024-07-25 03:03:17,943 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 03:03:17,991 INFO [upload_forcing] forecast2 2024-07-25 forcing files upload to orcinus-nowcast-agrif completed
2024-07-25 03:03:18,145 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 03:03:18,507 WARNING [get_onc_ctd] no SEVIP salinity data
2024-07-25 03:03:18,806 INFO [get_onc_ctd] 2024-07-24 ONC SEVIP CTD T&S file created
2024-07-25 03:03:18,809 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-07-25 03:03:18,908 WARNING [get_onc_ferry] No ONC TWDP TSG temperature,conductivity,salinity data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:18,912 WARNING [get_onc_ferry] No ONC TWDP TSG temperature data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:18,913 WARNING [get_onc_ferry] No ONC TWDP TSG conductivity data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:18,915 WARNING [get_onc_ferry] No ONC TWDP TSG salinity data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:18,917 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-07-24
2024-07-25 03:03:19,015 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-07-25 03:03:19,100 INFO [upload_forcing] forecast2 2024-07-25 forcing files upload to arbutus.cloud-nowcast completed
2024-07-25 03:03:19,103 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 03:03:19,338 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 03:03:19,746 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation,oxygen_corrected,temperature data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:19,752 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:19,753 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:19,755 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR temperature data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:19,756 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-07-24
2024-07-25 03:03:20,163 INFO [ping_erddap] running in process 765521
2024-07-25 03:03:20,164 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:20,165 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-25 03:03:20,166 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:20,168 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-07-25 03:03:20,169 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 03:03:20,386 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence,chlorophyll,turbidity data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:20,392 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:20,394 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL chlorophyll data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:20,395 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL turbidity data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:20,396 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-07-24
2024-07-25 03:03:20,539 INFO [make_forcing_links] running in process 765523
2024-07-25 03:03:20,540 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:20,541 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-25 03:03:20,543 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:21,668 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:21,672 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:21,674 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:21,675 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-07-24
2024-07-25 03:03:21,682 INFO [make_forcing_links] forecast2 2024-07-25 forcing file links on arbutus.cloud-nowcast created
2024-07-25 03:03:21,686 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-25 03:03:21,899 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 03:03:23,916 INFO [upload_forcing] forecast2 2024-07-25 forcing files upload to robot.graham completed
2024-07-25 03:03:23,921 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-25 03:03:24,055 INFO [run_NEMO] running in process 28107
2024-07-25 03:03:24,056 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:24,057 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-25 03:03:24,067 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:26,687 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/24jul24forecast2_2024-07-25T030324.185690-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-25 03:03:26,705 INFO [run_NEMO] forecast2 NEMO run for 2024-07-25 on arbutus.cloud-nowcast started
2024-07-25 03:03:26,710 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-25 03:03:26,911 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-25 03:03:28,900 INFO [watch_NEMO] running in process 28360
2024-07-25 03:03:28,901 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:03:28,902 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-25 03:03:28,903 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:03:28,975 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-25 03:03:38,949 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-07-24
2024-07-25 03:03:47,386 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-07-24
2024-07-25 03:03:55,538 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-07-24; substituting empty dataset
2024-07-25 03:03:55,539 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-07-24
2024-07-25 03:04:03,516 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-24; substituting empty dataset
2024-07-25 03:04:03,519 WARNING [get_onc_ferry] ONC TWDP TSG temperature data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,541 WARNING [get_onc_ferry] ONC TWDP TSG conductivity data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,604 WARNING [get_onc_ferry] ONC TWDP TSG salinity data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,621 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,638 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,656 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR temperature data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,674 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,691 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL chlorophyll data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,709 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL turbidity data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,726 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,744 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,846 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:04:03,863 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-25 03:05:13,147 INFO [get_onc_ferry] 2024-07-24 ONC TWDP ferry data file created
2024-07-25 03:05:13,151 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-07-25 03:05:13,358 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-07-25 03:05:14,635 INFO [ping_erddap] running in process 765546
2024-07-25 03:05:14,637 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:05:14,638 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-25 03:05:14,640 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:05:14,641 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-07-25 03:05:14,644 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 03:08:28,980 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17809659 = 2024-07-26 05:05:20 UTC, 17.0% complete
2024-07-25 03:13:28,980 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17810167 = 2024-07-26 10:44:00 UTC, 35.8% complete
2024-07-25 03:18:28,982 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17810673 = 2024-07-26 16:21:20 UTC, 54.5% complete
2024-07-25 03:23:28,983 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17811194 = 2024-07-26 22:08:40 UTC, 73.8% complete
2024-07-25 03:28:28,985 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17811694 = 2024-07-27 03:42:00 UTC, 92.4% complete
2024-07-25 03:33:28,989 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-07-25 03:33:28,993 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-25 03:33:29,343 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 03:33:29,351 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 03:33:29,355 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-25 03:33:30,847 INFO [download_results] running in process 765707
2024-07-25 03:33:30,848 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:33:30,849 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-25 03:33:30,850 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:33:30,851 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/24jul24 to /results/SalishSea/forecast2.202111
2024-07-25 03:33:32,051 INFO [make_ww3_wind_file] running in process 28582
2024-07-25 03:33:32,052 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:33:32,053 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-25 03:33:32,054 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:33:32,056 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-25 forecast2 run
2024-07-25 03:33:32,260 INFO [make_ww3_current_file] running in process 28584
2024-07-25 03:33:32,261 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:33:32,262 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-25 03:33:32,263 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:33:32,265 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-25 forecast2 run
2024-07-25 03:33:40,741 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-25 forecast2 run
2024-07-25 03:33:40,744 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-25 03:35:39,038 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-25 forecast2 run
2024-07-25 03:35:39,042 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-25 03:35:39,257 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-25'], host='arbutus.cloud-nowcast')
2024-07-25 03:35:47,312 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-25 03:35:47,522 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-25 03:35:56,870 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-25 03:40:56,972 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 75, 13.9% complete
2024-07-25 03:45:56,768 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 165, 30.6% complete
2024-07-25 03:48:00,247 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-25 03:48:00,612 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-01-26'], host='localhost')
2024-07-25 03:48:00,619 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 03:48:00,624 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-25 03:48:01,903 INFO [download_results] running in process 765750
2024-07-25 03:48:01,905 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:48:01,906 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-25 03:48:01,907 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:48:01,908 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jan19 to /ocean/sallen/timbucktwo/oxygen
2024-07-25 03:48:06,193 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-25 03:48:47,603 INFO [download_results] hindcast 2019-01-26 results files from optimum-hindcast downloaded
2024-07-25 03:48:47,613 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 03:48:47,820 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-01-26'], host='localhost')
2024-07-25 03:48:49,182 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-25 03:48:49,390 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-jan', 'robot.graham'], host='localhost')
2024-07-25 03:48:50,569 INFO [archive_tarball] running in process 765795
2024-07-25 03:48:50,570 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:48:50,572 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-07-25 03:48:50,573 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:48:50,574 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jan19.tar from /ocean/sallen/timbucktwo/oxygen/*jan19/
2024-07-25 03:50:56,360 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 255, 47.2% complete
2024-07-25 03:54:46,569 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jan19.index from /ocean/dlatorne/oxygen-jan19.tar
2024-07-25 03:54:46,652 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jan19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-25 03:55:56,477 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 351, 65.0% complete
2024-07-25 03:58:25,080 INFO [download_results] forecast2 2024-07-24 results files from arbutus.cloud-nowcast downloaded
2024-07-25 03:58:25,085 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-25 03:58:25,293 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-25 03:58:27,647 INFO [make_CHS_currents_file] running in process 765861
2024-07-25 03:58:27,648 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:58:27,649 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-25 03:58:27,651 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:59:03,990 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-24 for forecast2
2024-07-25 03:59:03,992 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-25 03:59:04,166 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-25 03:59:05,445 INFO [update_forecast_datasets] running in process 765886
2024-07-25 03:59:05,447 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 03:59:05,448 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2024-07-25 03:59:05,449 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-25 03:59:05,451 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/24jul24 run
2024-07-25 04:00:56,356 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 441, 81.7% complete
2024-07-25 04:01:37,335 INFO [archive_tarball] hindcast *jan19 results files archived to robot.graham
2024-07-25 04:01:37,339 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-25 04:03:23,939 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/24jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-25 04:03:23,940 INFO [update_forecast_datasets] nemo 2024-07-24 forecast2 rolling forecast datasets updated
2024-07-25 04:03:23,942 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-25 04:03:24,152 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-25 04:03:24,158 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-07-24'], host='localhost')
2024-07-25 04:03:24,163 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-25 04:03:25,462 INFO [ping_erddap] running in process 765975
2024-07-25 04:03:25,463 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:03:25,464 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-25 04:03:25,465 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:03:25,475 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-25 04:03:25,477 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 04:03:26,672 INFO [make_surface_current_tiles] running in process 765978
2024-07-25 04:03:26,673 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:03:26,674 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2024-07-25 04:03:26,675 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:03:26,677 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-24 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/24jul24
2024-07-25 04:03:28,729 INFO [make_plots] running in process 765977
2024-07-25 04:03:28,731 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:03:28,731 INFO [make_plots] publishing log messages to tcp://*:5564
2024-07-25 04:03:28,732 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:03:29,411 INFO [make_plots] preparing render list for 2024-07-24 NEMO forecast2 publish figures
2024-07-25 04:03:29,531 INFO [make_plots] added Website_thumbnail figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,532 INFO [make_plots] added Threshold_website figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,533 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,534 INFO [make_plots] added Boundary Bay figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,535 INFO [make_plots] added Campbell River figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,536 INFO [make_plots] added Cherry Point figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,537 INFO [make_plots] added Friday Harbor figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,538 INFO [make_plots] added Halfmoon Bay figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,539 INFO [make_plots] added Nanaimo figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,540 INFO [make_plots] added Neah Bay figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,541 INFO [make_plots] added New Westminster figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,541 INFO [make_plots] added Patricia Bay figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,542 INFO [make_plots] added Point Atkinson figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,543 INFO [make_plots] added Port Renfrew figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,543 INFO [make_plots] added Sand Heads figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,544 INFO [make_plots] added Sandy Cove figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,545 INFO [make_plots] added Squamish figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,545 INFO [make_plots] added Victoria figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,546 INFO [make_plots] added Woodwards Landing figure to 2024-07-24 NEMO forecast2 publish render list
2024-07-25 04:03:29,547 INFO [make_plots] starting to render nemo forecast2 publish 24jul24 figures
2024-07-25 04:05:56,356 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 531, 98.3% complete
2024-07-25 04:10:56,358 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-25 04:10:56,390 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-25 04:10:56,721 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 04:10:57,970 INFO [download_wwatch3_results] running in process 767301
2024-07-25 04:10:57,971 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:10:57,972 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-07-25 04:10:57,973 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:13:17,009 INFO [download_wwatch3_results] forecast2 2024-07-25 results files from arbutus.cloud-nowcast downloaded
2024-07-25 04:13:17,012 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-25 04:13:17,402 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 04:13:19,185 INFO [update_forecast_datasets] running in process 767679
2024-07-25 04:13:19,187 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:13:19,188 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-07-25 04:13:19,189 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:13:19,190 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/25jul24 run
2024-07-25 04:13:19,192 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/25jul24 run
2024-07-25 04:13:53,563 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/25jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-25 04:13:53,563 INFO [update_forecast_datasets] wwatch3 2024-07-25 forecast2 rolling forecast datasets updated
2024-07-25 04:13:53,566 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-25 04:13:53,975 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-25 04:13:55,524 INFO [ping_erddap] running in process 767808
2024-07-25 04:13:55,525 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:13:55,526 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-25 04:13:55,527 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:13:55,529 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-25 04:13:55,530 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-25 04:13:55,938 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-07-25'], host='localhost')
2024-07-25 04:14:01,260 INFO [make_plots] running in process 767816
2024-07-25 04:14:01,261 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:14:01,262 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-25 04:14:01,263 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:14:01,264 INFO [make_plots] preparing render list for 2024-07-25 WaveWatch3 forecast2 publish figures
2024-07-25 04:14:01,265 INFO [make_plots] added Halibut Bank figure to 2024-07-25 WaveWatch3 forecast2 publish render list
2024-07-25 04:14:01,265 INFO [make_plots] added Sentry Shoal figure to 2024-07-25 WaveWatch3 forecast2 publish render list
2024-07-25 04:14:01,266 INFO [make_plots] starting to render wwatch3 forecast2 publish 25jul24 figures
2024-07-25 04:14:01,789 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-25 04:14:10,714 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-25 04:14:17,178 INFO [make_plots] finished rendering wwatch3 forecast2 publish 25jul24 figures
2024-07-25 04:14:17,178 INFO [make_plots] wwatch3 publish plots for 2024-07-25 forecast2 completed
2024-07-25 04:14:17,181 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-25 04:15:05,038 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/24jul24
2024-07-25 04:16:37,449 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-25 04:16:37,450 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-25 04:17:12,438 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:18:57,948 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:18:58,309 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:19:15,609 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/24jul24
2024-07-25 04:19:15,617 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-24 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/24jul24
2024-07-25 04:19:15,619 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-24 forecast2 completed
2024-07-25 04:19:15,856 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-25 04:19:32,706 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:20:41,586 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:21:16,803 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:21:51,860 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:22:27,299 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:23:01,548 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:23:36,574 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:24:11,529 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:24:11,885 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:24:46,504 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:25:20,445 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-20 00:00:00Z to 2024-07-27 05:55:00Z
2024-07-25 04:25:55,426 INFO [make_plots] finished rendering nemo forecast2 publish 24jul24 figures
2024-07-25 04:25:55,429 INFO [make_plots] nemo publish plots for 2024-07-24 forecast2 completed
2024-07-25 04:25:55,435 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-25 04:25:55,762 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-25 04:25:58,729 INFO [make_feeds] running in process 768082
2024-07-25 04:25:58,730 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:25:58,731 INFO [make_feeds] publishing log messages to tcp://*:5562
2024-07-25 04:25:58,733 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:26:26,897 INFO [make_feeds] ATOM feeds for 2024-07-24 forecast2 run completed
2024-07-25 04:26:26,900 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-25 04:26:27,192 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-07-25 04:26:28,478 INFO [clear_checklist] running in process 768108
2024-07-25 04:26:28,479 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:26:28,481 INFO [clear_checklist] publishing log messages to tcp://*:5562
2024-07-25 04:26:28,482 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:26:28,483 INFO [clear_checklist] requesting that manager clear system state checklist
2024-07-25 04:26:28,486 INFO [manager] writing checklist to log file
2024-07-25 04:26:28,537 INFO [manager] checklist cleared
2024-07-25 04:26:28,539 INFO [clear_checklist] nowcast system checklist cleared
2024-07-25 04:26:28,542 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-07-25 04:26:29,819 INFO [rotate_logs] running in process 768113
2024-07-25 04:26:29,821 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-25 04:26:29,822 INFO [rotate_logs] publishing log messages to tcp://*:5562
2024-07-25 04:26:29,823 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-07-25 04:26:29,824 INFO [rotate_logs] rotating log files