2024-04-22 04:46:09,359 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-04-22 04:46:09,366 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-04-22 04:46:09,371 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-04-22 04:46:09,373 INFO [rotate_logs] log files rotated
2024-04-22 04:46:09,379 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-04-22 05:33:39,703 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 05:33:39,894 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-03-01'], host='localhost')
2024-04-22 05:33:39,900 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 05:33:39,905 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 05:33:40,900 INFO [download_results] running in process 2099339
2024-04-22 05:33:40,902 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 05:33:40,903 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-22 05:33:40,904 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 05:33:40,905 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01mar20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 05:33:45,101 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 05:34:29,472 INFO [download_results] hindcast 2020-03-01 results files from optimum-hindcast downloaded
2024-04-22 05:34:29,488 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 05:34:29,495 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-03-01'], host='localhost')
2024-04-22 05:34:30,529 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 06:58:49,627 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 06:58:49,832 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-03-06'], host='localhost')
2024-04-22 06:58:49,838 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 06:58:49,841 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 06:58:50,799 INFO [download_results] running in process 2099681
2024-04-22 06:58:50,801 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 06:58:50,802 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-22 06:58:50,803 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 06:58:50,804 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06mar20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 06:58:54,952 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 06:59:39,476 INFO [download_results] hindcast 2020-03-06 results files from optimum-hindcast downloaded
2024-04-22 06:59:39,491 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 06:59:39,501 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-03-06'], host='localhost')
2024-04-22 06:59:40,507 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 08:23:59,501 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 08:23:59,698 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-03-11'], host='localhost')
2024-04-22 08:23:59,705 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 08:23:59,709 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 08:24:00,651 INFO [download_results] running in process 2100252
2024-04-22 08:24:00,652 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 08:24:00,654 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-22 08:24:00,655 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 08:24:00,656 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11mar20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 08:24:04,845 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 08:24:49,418 INFO [download_results] hindcast 2020-03-11 results files from optimum-hindcast downloaded
2024-04-22 08:24:49,432 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 08:24:49,440 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-03-11'], host='localhost')
2024-04-22 08:24:50,454 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 09:03:58,095 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240422/12/
2024-04-22 09:03:58,096 INFO [collect_weather] 2024-04-22 2.5km weather forecast 12 collection complete
2024-04-22 09:03:58,099 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-22 09:03:58,299 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 09:03:58,304 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 09:03:58,309 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 09:03:58,313 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 09:03:58,317 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-04-22 09:03:58,322 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-04-22 09:03:58,327 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-04-22 09:03:58,332 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-04-22 09:03:58,335 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-04-22 09:03:59,673 INFO [collect_NeahBay_ssh] running in process 2100671
2024-04-22 09:03:59,674 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:03:59,675 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2024-04-22 09:03:59,676 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:03:59,676 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-04-22
2024-04-22 09:03:59,678 INFO [collect_weather] running in process 2100673
2024-04-22 09:03:59,679 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:03:59,679 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-04-22 09:03:59,680 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:03:59,725 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-04-22 09:03:59,764 INFO [download_live_ocean] running in process 2100672
2024-04-22 09:03:59,765 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:03:59,765 INFO [download_live_ocean] publishing log messages to tcp://*:5566
2024-04-22 09:03:59,767 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:03:59,767 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-04-22
2024-04-22 09:04:00,212 INFO [make_turbidity_file] running in process 2100670
2024-04-22 09:04:00,213 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:00,214 INFO [make_turbidity_file] publishing log messages to tcp://*:5567
2024-04-22 09:04:00,215 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:00,216 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-04-22
2024-04-22 09:04:00,328 INFO [collect_river_data] running in process 2100669
2024-04-22 09:04:00,330 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:00,331 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-04-22 09:04:00,332 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:00,333 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-04-21
2024-04-22 09:04:00,392 INFO [collect_river_data] running in process 2100667
2024-04-22 09:04:00,393 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:00,395 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-04-22 09:04:00,396 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:00,396 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-04-21
2024-04-22 09:04:00,399 INFO [collect_river_data] running in process 2100665
2024-04-22 09:04:00,400 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:00,401 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-22 09:04:00,402 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:00,403 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-04-21
2024-04-22 09:04:00,494 INFO [collect_river_data] running in process 2100668
2024-04-22 09:04:00,496 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:00,497 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-04-22 09:04:00,498 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:00,499 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-04-21
2024-04-22 09:04:00,517 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-04-21&endDT=2024-04-21 "HTTP/1.1 200 OK"
2024-04-22 09:04:00,571 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-04-21&endDT=2024-04-21 "HTTP/1.1 200 OK"
2024-04-22 09:04:00,645 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-04-22 09:04:00,646 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-04-21 completed
2024-04-22 09:04:00,648 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 09:04:00,649 INFO [crop_gribs] running in process 2100674
2024-04-22 09:04:00,650 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:00,652 INFO [crop_gribs] publishing log messages to tcp://*:5572
2024-04-22 09:04:00,652 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:00,653 INFO [crop_gribs] cropping 2024-04-22 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-04-22 09:04:00,655 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-04-22 09:04:00,656 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-04-21 completed
2024-04-22 09:04:00,661 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 09:04:00,663 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240422/18/
2024-04-22 09:04:00,729 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-04-21&endDT=2024-04-21 "HTTP/1.1 200 OK"
2024-04-22 09:04:00,745 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-04-22 09:04:00,746 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-04-21 completed
2024-04-22 09:04:00,748 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 09:04:00,830 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-04-21&endDT=2024-04-21 "HTTP/1.1 200 OK"
2024-04-22 09:04:00,834 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-04-22 09:04:00,835 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-04-21 completed
2024-04-22 09:04:00,837 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 09:04:01,144 INFO [collect_NeahBay_ssh] 2024-04-22T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-04-22 09:04:01,147 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-22 09:04:01,160 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 09:04:02,995 INFO [make_turbidity_file] 2024-04-22 Fraser River turbidity file creation complete
2024-04-22 09:04:02,997 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-04-22 09:04:03,654 INFO [make_ssh_files] running in process 2100824
2024-04-22 09:04:03,655 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:03,656 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2024-04-22 09:04:03,657 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:03,658 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-04-22 NOAA Neah Bay 06Z observation and forecast values
2024-04-22 09:04:07,862 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d18.nc
2024-04-22 09:04:08,048 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d19.nc
2024-04-22 09:04:08,229 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d20.nc
2024-04-22 09:04:08,409 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d21.nc
2024-04-22 09:04:08,638 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d22.nc
2024-04-22 09:04:08,817 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d23.nc
2024-04-22 09:04:08,992 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d24.nc
2024-04-22 09:04:09,170 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d25.nc
2024-04-22 09:04:09,283 INFO [make_ssh_files] sea surface height boundary file for 2024-04-22 nowcast run created
2024-04-22 09:04:09,288 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-22 09:04:09,298 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-22 09:04:09,303 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-22 09:04:10,321 INFO [make_runoff_file] running in process 2100858
2024-04-22 09:04:10,324 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:10,325 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2024-04-22 09:04:10,326 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:10,352 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-22 09:04:10,437 INFO [make_runoff_file] 2024-04-22 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-22 09:04:10,441 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-22 09:04:11,043 INFO [make_v202111_runoff_file] running in process 2100856
2024-04-22 09:04:11,045 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:04:11,046 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5563
2024-04-22 09:04:11,047 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:04:11,048 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-21
2024-04-22 09:04:13,520 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-21: /results/forcing/rivers/R202108Dailies_y2024m04d21.nc
2024-04-22 09:04:13,521 INFO [make_v202111_runoff_file] 2024-04-21 runoff file creation completed
2024-04-22 09:04:13,523 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-22 09:12:19,951 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240422/12/
2024-04-22 09:12:19,952 INFO [crop_gribs] 2024-04-22 12 GRIBs cropping complete
2024-04-22 09:12:19,955 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-22 09:12:20,171 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-04-22 09:12:22,108 INFO [grib_to_netcdf] running in process 2100955
2024-04-22 09:12:22,110 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:12:22,111 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2024-04-22 09:12:22,112 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:12:22,113 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-22 nowcast and forecast runs
2024-04-22 09:12:36,032 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m04d22.nc
2024-04-22 09:12:46,580 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d23.nc
2024-04-22 09:12:52,487 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d24.nc
2024-04-22 09:12:52,505 INFO [grib_to_netcdf] 2024-04-22 NEMO-atmos forcing file for nowcast+ created
2024-04-22 09:12:52,508 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-22 09:49:09,441 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 09:49:09,656 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-03-16'], host='localhost')
2024-04-22 09:49:09,663 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 09:49:09,668 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 09:49:10,705 INFO [download_results] running in process 2101117
2024-04-22 09:49:10,706 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 09:49:10,707 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-22 09:49:10,708 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 09:49:10,709 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16mar20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 09:49:14,923 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 09:50:00,330 INFO [download_results] hindcast 2020-03-16 results files from optimum-hindcast downloaded
2024-04-22 09:50:00,345 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 09:50:00,353 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-03-16'], host='localhost')
2024-04-22 09:50:01,388 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 11:14:19,372 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 11:14:19,711 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-03-21'], host='localhost')
2024-04-22 11:14:19,717 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 11:14:19,721 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 11:14:20,770 INFO [download_results] running in process 2101332
2024-04-22 11:14:20,771 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 11:14:20,772 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-22 11:14:20,773 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 11:14:20,774 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21mar20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 11:14:24,917 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 11:15:09,643 INFO [download_results] hindcast 2020-03-21 results files from optimum-hindcast downloaded
2024-04-22 11:15:09,659 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 11:15:09,672 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-03-21'], host='localhost')
2024-04-22 11:15:10,668 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 12:04:02,864 ERROR [download_live_ocean] giving up after 37 attempts: 404 Client Error: Not Found for url: https://liveocean.apl.uw.edu/output/f2024.04.22/ubc_done.txt for https://liveocean.apl.uw.edu/output/f2024.04.22/ubc_done.txt
2024-04-22 12:04:03,123 CRITICAL [download_live_ocean] 2024-04-22 Live Ocean file for Salish Sea western boundary download failed
2024-04-22 12:10:49,702 INFO [download_live_ocean] running in process 2101684
2024-04-22 12:10:49,703 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:10:49,705 INFO [download_live_ocean] publishing log messages to tcp://*:5563
2024-04-22 12:10:49,706 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:10:49,707 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-04-22
2024-04-22 12:35:50,669 INFO [download_live_ocean] downloaded 10129091 bytes from https://liveocean.apl.uw.edu/output/f2024.04.22/ubc.nc to /results/forcing/LiveOcean/downloaded/20240422/low_passed_UBC.nc
2024-04-22 12:35:50,670 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-04-22 12:35:51,715 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240422/low_passed_UBC.nc
2024-04-22 12:35:51,716 INFO [download_live_ocean] 2024-04-22 Live Ocean file for Salish Sea western boundary downloaded
2024-04-22 12:35:51,719 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-04-22 12:35:51,914 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-04-22'], host='localhost')
2024-04-22 12:35:54,187 INFO [make_live_ocean_files] running in process 2101875
2024-04-22 12:35:54,188 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:35:54,189 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563
2024-04-22 12:35:54,191 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:35:54,192 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-04-22 Live Ocean run
2024-04-22 12:36:35,629 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2024-04-22 12:36:35,630 INFO [make_live_ocean_files] 2024-04-22 Live Ocean western boundary conditions files created
2024-04-22 12:36:35,633 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-04-22 12:36:35,647 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-04-22 12:36:35,649 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-04-22 12:36:35,653 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-04-22 12:36:35,657 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-04-22 12:36:35,661 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-04-22 12:36:36,466 INFO [ping_erddap] running in process 2101899
2024-04-22 12:36:36,468 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:36:36,469 INFO [ping_erddap] publishing log messages to tcp://*:5563
2024-04-22 12:36:36,470 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:36:36,472 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-04-22 12:36:36,474 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-22 12:36:36,575 INFO [upload_forcing] running in process 2101901
2024-04-22 12:36:36,576 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:36:36,577 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-04-22 12:36:36,578 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:36:36,604 INFO [upload_forcing] running in process 2101903
2024-04-22 12:36:36,605 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:36:36,607 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-22 12:36:36,608 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:36:36,611 INFO [upload_forcing] running in process 2101902
2024-04-22 12:36:36,612 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:36:36,612 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-22 12:36:36,613 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:36:36,619 INFO [upload_forcing] running in process 2101904
2024-04-22 12:36:36,620 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:36:36,621 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-22 12:36:36,621 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:36:38,793 INFO [upload_forcing] nowcast+ 2024-04-22 forcing files upload to optimum-hindcast completed
2024-04-22 12:36:38,796 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 12:36:38,799 INFO [upload_forcing] nowcast+ 2024-04-22 forcing files upload to orcinus-nowcast-agrif completed
2024-04-22 12:36:38,810 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 12:36:46,050 INFO [upload_forcing] nowcast+ 2024-04-22 forcing files upload to robot.graham completed
2024-04-22 12:36:46,055 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 12:36:49,737 INFO [upload_forcing] nowcast+ 2024-04-22 forcing files upload to arbutus.cloud-nowcast completed
2024-04-22 12:36:49,742 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 12:36:49,758 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 12:36:50,799 INFO [make_forcing_links] running in process 2101939
2024-04-22 12:36:50,800 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:36:50,801 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2024-04-22 12:36:50,803 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:36:51,967 INFO [make_forcing_links] nowcast+ 2024-04-22 forcing file links on arbutus.cloud-nowcast created
2024-04-22 12:36:51,972 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-22 12:36:51,988 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 12:36:54,007 INFO [run_NEMO] running in process 8715
2024-04-22 12:36:54,009 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:36:54,010 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-22 12:36:54,132 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:36:54,852 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/22apr24nowcast_2024-04-22T123654.058320-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-22 12:36:54,873 INFO [run_NEMO] nowcast NEMO run for 2024-04-22 on arbutus.cloud-nowcast started
2024-04-22 12:36:54,878 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-22 12:36:54,891 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-22 12:36:56,773 INFO [watch_NEMO] running in process 8969
2024-04-22 12:36:56,774 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:36:56,776 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-22 12:36:56,777 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:36:56,839 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-22 12:41:56,840 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17604422 = 2024-04-22 04:40:40 UTC, 19.5% complete
2024-04-22 12:46:56,841 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17604908 = 2024-04-22 10:04:40 UTC, 42.0% complete
2024-04-22 12:51:56,841 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17605409 = 2024-04-22 15:38:40 UTC, 65.2% complete
2024-04-22 12:56:56,843 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17605889 = 2024-04-22 20:58:40 UTC, 87.4% complete
2024-04-22 12:59:31,220 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 12:59:31,427 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-03-26'], host='localhost')
2024-04-22 12:59:31,434 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 12:59:31,438 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 12:59:32,389 INFO [download_results] running in process 2102319
2024-04-22 12:59:32,391 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 12:59:32,392 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-22 12:59:32,393 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 12:59:32,394 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26mar20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 12:59:36,558 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 13:00:28,453 INFO [download_results] hindcast 2020-03-26 results files from optimum-hindcast downloaded
2024-04-22 13:00:28,469 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 13:00:28,484 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-03-26'], host='localhost')
2024-04-22 13:00:29,430 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 13:00:29,448 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2020-mar', 'robot.graham'], host='localhost')
2024-04-22 13:00:30,404 INFO [archive_tarball] running in process 2102365
2024-04-22 13:00:30,405 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:00:30,407 INFO [archive_tarball] publishing log messages to tcp://*:5563
2024-04-22 13:00:30,408 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:00:30,409 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar20.tar from /ocean/sallen/timbucktwo/oxygen/*mar20/
2024-04-22 13:01:56,848 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-04-22 13:01:56,851 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-22 13:01:57,071 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:01:57,079 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:01:58,269 INFO [make_forcing_links] running in process 2102389
2024-04-22 13:01:58,271 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:01:58,272 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-04-22 13:01:58,273 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:01:58,275 INFO [download_results] running in process 2102390
2024-04-22 13:01:58,276 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:01:58,277 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-22 13:01:58,278 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:01:58,279 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/22apr24 to /results/SalishSea/nowcast-blue.202111
2024-04-22 13:01:59,175 INFO [make_forcing_links] ssh 2024-04-22 forcing file links on arbutus.cloud-nowcast created
2024-04-22 13:01:59,180 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-22 13:01:59,195 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 13:02:01,286 INFO [run_NEMO] running in process 9939
2024-04-22 13:02:01,286 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:02:01,288 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-22 13:02:01,289 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:02:02,195 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/22apr24forecast_2024-04-22T130201.360477-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-22 13:02:02,215 INFO [run_NEMO] forecast NEMO run for 2024-04-22 on arbutus.cloud-nowcast started
2024-04-22 13:02:02,219 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-22 13:02:02,235 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-22 13:02:04,218 INFO [watch_NEMO] running in process 10191
2024-04-22 13:02:04,220 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:02:04,221 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-22 13:02:04,223 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:02:04,287 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-22 13:06:45,411 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-mar20.index from /ocean/dlatorne/oxygen-mar20.tar
2024-04-22 13:06:45,577 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-mar20.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-22 13:07:04,473 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17606610 = 2024-04-23 04:59:20 UTC, 13.9% complete
2024-04-22 13:12:04,376 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17607074 = 2024-04-23 10:08:40 UTC, 28.2% complete
2024-04-22 13:16:09,913 INFO [archive_tarball] hindcast *mar20 results files archived to robot.graham
2024-04-22 13:16:09,917 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-22 13:17:04,467 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17607524 = 2024-04-23 15:08:40 UTC, 42.1% complete
2024-04-22 13:22:04,333 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17607989 = 2024-04-23 20:18:40 UTC, 56.4% complete
2024-04-22 13:22:10,233 INFO [download_results] nowcast 2024-04-22 results files from arbutus.cloud-nowcast downloaded
2024-04-22 13:22:10,239 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 13:22:10,259 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:22:10,266 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-04-21'], host='localhost')
2024-04-22 13:22:10,271 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:22:12,739 INFO [make_CHS_currents_file] running in process 2102536
2024-04-22 13:22:12,740 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:22:12,741 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2024-04-22 13:22:12,742 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:22:14,695 INFO [make_plots] running in process 2102533
2024-04-22 13:22:14,696 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:22:14,697 INFO [make_plots] publishing log messages to tcp://*:5565
2024-04-22 13:22:14,698 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:22:14,701 INFO [make_plots] running in process 2102535
2024-04-22 13:22:14,703 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:22:14,704 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-22 13:22:14,705 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:22:16,174 INFO [make_plots] preparing render list for 2024-04-21 NEMO nowcast-blue comparison figures
2024-04-22 13:22:16,178 INFO [make_plots] preparing render list for 2024-04-22 NEMO nowcast-blue research figures
2024-04-22 13:22:17,145 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-blue research render list
2024-04-22 13:22:17,793 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-blue research render list
2024-04-22 13:22:17,794 INFO [make_plots] added Currents_sections_and_surface figure to 2024-04-22 NEMO nowcast-blue research render list
2024-04-22 13:22:17,796 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-04-22 NEMO nowcast-blue research render list
2024-04-22 13:22:17,797 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-04-22 NEMO nowcast-blue research render list
2024-04-22 13:22:17,798 INFO [make_plots] starting to render nemo nowcast research 22apr24 figures
2024-04-22 13:22:43,282 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-22 for nowcast
2024-04-22 13:22:43,285 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-22 13:22:57,474 INFO [make_plots] added SH_wind figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,474 INFO [make_plots] added Compare_VENUS_East figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,478 INFO [make_plots] added Compare_VENUS_Central figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,479 INFO [make_plots] added Central_ADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,480 INFO [make_plots] added Central_depavADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,481 INFO [make_plots] added Central_timeavADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,482 INFO [make_plots] added East_ADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,483 INFO [make_plots] added East_depavADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,484 INFO [make_plots] added East_timeavADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,485 INFO [make_plots] added ddl_ADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,486 INFO [make_plots] added ddl_depavADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,486 INFO [make_plots] added ddl_timeavADCP figure to 2024-04-21 NEMO nowcast-blue comparison render list
2024-04-22 13:22:57,487 INFO [make_plots] starting to render nemo nowcast comparison 21apr24 figures
2024-04-22 13:23:33,187 INFO [make_plots] finished rendering nemo nowcast research 22apr24 figures
2024-04-22 13:23:33,190 INFO [make_plots] nemo research plots for 2024-04-22 nowcast completed
2024-04-22 13:23:33,208 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-22 13:24:54,315 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-22 13:24:54,784 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-22 13:24:55,254 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-22 13:24:55,715 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-22 13:24:56,173 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-22 13:24:56,630 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-22 13:24:57,086 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-22 13:24:57,544 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-22 13:24:58,016 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-22 13:24:58,018 INFO [make_plots] finished rendering nemo nowcast comparison 21apr24 figures
2024-04-22 13:24:58,129 INFO [make_plots] nemo comparison plots for 2024-04-21 nowcast completed
2024-04-22 13:24:58,133 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-22 13:27:04,252 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17608435 = 2024-04-24 01:16:00 UTC, 70.2% complete
2024-04-22 13:32:04,361 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17608930 = 2024-04-24 06:46:00 UTC, 85.5% complete
2024-04-22 13:37:04,257 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17609397 = 2024-04-24 11:57:20 UTC, 99.9% complete
2024-04-22 13:42:04,176 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-22 13:42:04,213 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-04-22 13:42:04,390 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 13:42:04,397 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 13:42:04,401 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:42:04,408 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:42:04,413 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:42:04,417 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:42:04,421 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:42:05,717 INFO [upload_forcing] running in process 2102672
2024-04-22 13:42:05,719 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:05,720 INFO [upload_forcing] publishing log messages to tcp://*:5563
2024-04-22 13:42:05,721 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:05,722 INFO [download_results] running in process 2102673
2024-04-22 13:42:05,724 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:05,725 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-22 13:42:05,726 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:05,727 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/22apr24 to /results/SalishSea/forecast.202111
2024-04-22 13:42:05,730 INFO [upload_forcing] running in process 2102669
2024-04-22 13:42:05,731 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:05,733 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-22 13:42:05,734 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:05,743 INFO [upload_forcing] running in process 2102671
2024-04-22 13:42:05,744 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:05,745 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-22 13:42:05,746 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:05,775 INFO [upload_forcing] running in process 2102670
2024-04-22 13:42:05,777 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:05,778 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-22 13:42:05,780 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:06,395 INFO [upload_forcing] turbidity 2024-04-22 forcing files upload to optimum-hindcast completed
2024-04-22 13:42:06,400 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 13:42:06,500 INFO [upload_forcing] turbidity 2024-04-22 forcing files upload to orcinus-nowcast-agrif completed
2024-04-22 13:42:06,505 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 13:42:06,527 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:42:06,887 INFO [upload_forcing] turbidity 2024-04-22 forcing files upload to arbutus.cloud-nowcast completed
2024-04-22 13:42:06,891 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 13:42:06,912 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:42:07,641 INFO [make_forcing_links] running in process 2102704
2024-04-22 13:42:07,642 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:07,643 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2024-04-22 13:42:07,645 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:08,046 INFO [make_forcing_links] running in process 2102706
2024-04-22 13:42:08,048 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:08,049 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2024-04-22 13:42:08,050 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:08,485 INFO [make_forcing_links] nowcast-agrif 2024-04-22 forcing file links on orcinus-nowcast-agrif created
2024-04-22 13:42:08,490 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-22 13:42:08,511 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 13:42:09,721 INFO [run_NEMO_agrif] running in process 2102720
2024-04-22 13:42:09,722 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:09,724 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5563
2024-04-22 13:42:09,725 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:09,934 INFO [make_forcing_links] nowcast-green 2024-04-22 forcing file links on arbutus.cloud-nowcast created
2024-04-22 13:42:09,939 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-22 13:42:09,960 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 13:42:10,808 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-22 13:42:12,132 INFO [run_NEMO] running in process 11932
2024-04-22 13:42:12,134 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:12,135 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-22 13:42:12,136 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:12,951 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-22 13:42:12,973 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-04-22 13:42:13,023 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/22apr24nowcast-green_2024-04-22T134212.231678-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-22 13:42:13,027 INFO [run_NEMO] nowcast-green NEMO run for 2024-04-22 on arbutus.cloud-nowcast started
2024-04-22 13:42:14,948 INFO [watch_NEMO] running in process 12185
2024-04-22 13:42:14,949 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:14,950 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-22 13:42:14,952 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:15,003 INFO [upload_forcing] turbidity 2024-04-22 forcing files upload to robot.graham completed
2024-04-22 13:42:15,008 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 13:42:15,026 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-22 13:42:26,528 INFO [run_NEMO_agrif] job id for 22apr24nowcast-agrif: 11092903.orca2.ibb
2024-04-22 13:42:26,530 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-04-22 queued on orcinus-nowcast-agrif
2024-04-22 13:42:26,535 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-04-22 13:42:26,559 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11092903.orca2.ibb'], host='localhost')
2024-04-22 13:42:27,712 INFO [watch_NEMO_agrif] running in process 2102739
2024-04-22 13:42:27,713 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:27,715 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5563
2024-04-22 13:42:27,716 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:28,551 INFO [watch_NEMO_agrif] watching 22apr24nowcast-agrif job 11092903 on orcinus-nowcast-agrif
2024-04-22 13:42:28,789 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif job 11092903 is queued on orcinus-nowcast-agrif
2024-04-22 13:42:49,438 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-22 13:42:49,466 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 13:42:51,851 INFO [run_ww3] running in process 12275
2024-04-22 13:42:51,852 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:51,854 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-22 13:42:51,855 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:51,859 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-22T134251.982941-0700
2024-04-22 13:42:51,861 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-22T134251.982941-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-22 13:42:51,875 INFO [run_ww3] nowcast WaveWatch3 run for 2024-04-22 on arbutus.cloud-nowcast started
2024-04-22 13:42:51,879 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-22 13:42:51,907 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-22 13:42:54,284 INFO [watch_ww3] running in process 12361
2024-04-22 13:42:54,288 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 13:42:54,290 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-04-22 13:42:54,291 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-22 13:42:54,376 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-22 13:43:29,851 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-04-22 13:47:15,050 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17604158 = 2024-04-22 01:44:40 UTC, 7.3% complete
2024-04-22 13:47:54,372 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 57, 13.2% complete
2024-04-22 13:48:30,341 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7625004 = 2024-04-22 02:15:20 UTC, 9.4% complete
2024-04-22 13:52:15,609 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17604334 = 2024-04-22 03:42:00 UTC, 15.4% complete
2024-04-22 13:52:54,356 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 129, 29.9% complete
2024-04-22 13:53:30,799 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7625213 = 2024-04-22 04:34:40 UTC, 19.1% complete
2024-04-22 13:57:15,064 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17604511 = 2024-04-22 05:40:00 UTC, 23.6% complete
2024-04-22 13:57:54,431 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete
2024-04-22 13:58:31,255 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7625424 = 2024-04-22 06:55:20 UTC, 28.9% complete
2024-04-22 14:02:15,010 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17604688 = 2024-04-22 07:38:00 UTC, 31.8% complete
2024-04-22 14:02:54,309 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 261, 60.4% complete
2024-04-22 14:03:31,742 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7625632 = 2024-04-22 09:14:00 UTC, 38.5% complete
2024-04-22 14:07:15,254 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17604865 = 2024-04-22 09:36:00 UTC, 40.0% complete
2024-04-22 14:07:54,499 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 324, 75.0% complete
2024-04-22 14:08:32,224 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7625843 = 2024-04-22 11:34:40 UTC, 48.3% complete
2024-04-22 14:11:20,436 INFO [download_results] forecast 2024-04-22 results files from arbutus.cloud-nowcast downloaded
2024-04-22 14:11:20,441 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 14:11:20,464 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:11:23,325 INFO [make_CHS_currents_file] running in process 2102843
2024-04-22 14:11:23,326 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:11:23,327 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-04-22 14:11:23,329 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:12:04,954 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-22 for forecast
2024-04-22 14:12:04,957 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-22 14:12:04,981 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:12:05,955 INFO [update_forecast_datasets] running in process 2102868
2024-04-22 14:12:05,957 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:12:05,958 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-04-22 14:12:05,959 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:12:05,961 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/22apr24 run
2024-04-22 14:12:06,279 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/22apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-22 14:12:06,281 INFO [update_forecast_datasets] nemo 2024-04-22 forecast rolling forecast datasets updated
2024-04-22 14:12:06,283 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-22 14:12:06,305 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-22 14:12:06,310 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:12:06,315 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:12:07,312 INFO [ping_erddap] running in process 2102873
2024-04-22 14:12:07,313 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:12:07,314 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-04-22 14:12:07,315 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:12:07,328 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-22 14:12:07,331 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-22 14:12:08,749 INFO [make_surface_current_tiles] running in process 2102876
2024-04-22 14:12:08,750 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:12:08,752 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-04-22 14:12:08,753 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:12:08,755 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-22 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/22apr24
2024-04-22 14:12:11,032 INFO [make_plots] running in process 2102875
2024-04-22 14:12:11,034 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:12:11,035 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-22 14:12:11,036 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:12:12,571 INFO [make_plots] preparing render list for 2024-04-22 NEMO forecast publish figures
2024-04-22 14:12:12,769 INFO [make_plots] added Website_thumbnail figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,770 INFO [make_plots] added Threshold_website figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,771 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,773 INFO [make_plots] added Boundary Bay figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,774 INFO [make_plots] added Campbell River figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,775 INFO [make_plots] added Cherry Point figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,776 INFO [make_plots] added Friday Harbor figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,778 INFO [make_plots] added Halfmoon Bay figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,779 INFO [make_plots] added Nanaimo figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,780 INFO [make_plots] added Neah Bay figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,781 INFO [make_plots] added New Westminster figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,782 INFO [make_plots] added Patricia Bay figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,783 INFO [make_plots] added Point Atkinson figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,784 INFO [make_plots] added Port Renfrew figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,785 INFO [make_plots] added Sand Heads figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,786 INFO [make_plots] added Sandy Cove figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,787 INFO [make_plots] added Squamish figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,788 INFO [make_plots] added Victoria figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,789 INFO [make_plots] added Woodwards Landing figure to 2024-04-22 NEMO forecast publish render list
2024-04-22 14:12:12,790 INFO [make_plots] starting to render nemo forecast publish 22apr24 figures
2024-04-22 14:12:15,222 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17605041 = 2024-04-22 11:33:20 UTC, 48.2% complete
2024-04-22 14:12:54,534 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 387, 89.6% complete
2024-04-22 14:13:32,674 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7626053 = 2024-04-22 13:54:40 UTC, 58.0% complete
2024-04-22 14:17:15,199 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17605216 = 2024-04-22 13:30:00 UTC, 56.3% complete
2024-04-22 14:17:54,574 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-22 14:17:54,577 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-22 14:17:54,835 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:17:54,849 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 14:17:56,462 INFO [download_wwatch3_results] running in process 2103871
2024-04-22 14:17:56,463 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:17:56,465 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2024-04-22 14:17:56,466 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:17:57,410 INFO [run_ww3] running in process 12994
2024-04-22 14:17:57,411 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:17:57,413 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-22 14:17:57,414 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:17:57,419 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-22T141757.374705-0700
2024-04-22 14:17:57,421 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-22T141757.374705-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-22 14:17:57,442 INFO [run_ww3] forecast WaveWatch3 run for 2024-04-22 on arbutus.cloud-nowcast started
2024-04-22 14:17:57,446 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-22 14:17:57,497 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-22 14:17:59,837 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-22 14:18:33,156 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7626260 = 2024-04-22 16:12:40 UTC, 67.6% complete
2024-04-22 14:19:58,141 INFO [download_wwatch3_results] nowcast 2024-04-22 results files from arbutus.cloud-nowcast downloaded
2024-04-22 14:19:58,148 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-22 14:22:15,173 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17605393 = 2024-04-22 15:28:00 UTC, 64.5% complete
2024-04-22 14:22:59,803 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 57, 8.8% complete
2024-04-22 14:23:33,642 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7626470 = 2024-04-22 18:32:40 UTC, 77.3% complete
2024-04-22 14:24:41,078 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 14:24:41,310 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-04-01'], host='localhost')
2024-04-22 14:24:41,323 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 14:24:41,337 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 14:24:43,110 INFO [download_results] running in process 2104806
2024-04-22 14:24:43,112 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:24:43,113 INFO [download_results] publishing log messages to tcp://*:5567
2024-04-22 14:24:43,114 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:24:43,115 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01apr20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 14:24:47,611 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 14:26:04,273 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/22apr24
2024-04-22 14:26:08,610 INFO [download_results] hindcast 2020-04-01 results files from optimum-hindcast downloaded
2024-04-22 14:26:08,627 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 14:26:08,658 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-04-01'], host='localhost')
2024-04-22 14:26:10,030 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 14:27:15,162 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17605569 = 2024-04-22 17:25:20 UTC, 72.6% complete
2024-04-22 14:27:59,797 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 129, 19.9% complete
2024-04-22 14:28:34,095 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7626681 = 2024-04-22 20:53:20 UTC, 87.1% complete
2024-04-22 14:30:00,245 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/22apr24
2024-04-22 14:30:00,253 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-22 forecast into /results/nowcast-sys/figures/surface_currents/forecast/22apr24
2024-04-22 14:30:00,255 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-22 forecast completed
2024-04-22 14:30:00,580 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-22 14:30:16,773 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-22 14:30:16,774 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-22 14:30:52,365 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:32:15,174 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17605745 = 2024-04-22 19:22:40 UTC, 80.8% complete
2024-04-22 14:32:42,671 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:32:43,079 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:32:59,810 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 207, 31.9% complete
2024-04-22 14:33:18,126 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:33:34,580 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7626888 = 2024-04-22 23:11:20 UTC, 96.7% complete
2024-04-22 14:34:28,894 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:35:04,112 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:35:40,755 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:36:17,106 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:36:53,016 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:37:15,185 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17605921 = 2024-04-22 21:20:00 UTC, 88.9% complete
2024-04-22 14:37:29,508 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:37:59,820 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 282, 43.5% complete
2024-04-22 14:38:06,755 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:38:07,157 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:38:35,012 INFO [watch_NEMO_agrif] 22apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7626960 = 2024-04-22 23:59:20 UTC, 100.0% complete
2024-04-22 14:38:43,153 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:39:19,857 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-17 00:00:00Z to 2024-04-24 11:55:00Z
2024-04-22 14:39:56,705 INFO [make_plots] finished rendering nemo forecast publish 22apr24 figures
2024-04-22 14:39:56,708 INFO [make_plots] nemo publish plots for 2024-04-22 forecast completed
2024-04-22 14:39:56,717 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-22 14:39:56,897 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:39:59,882 INFO [make_feeds] running in process 2105104
2024-04-22 14:39:59,883 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:39:59,884 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-04-22 14:39:59,885 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:40:28,219 INFO [make_feeds] ATOM feeds for 2024-04-22 forecast run completed
2024-04-22 14:40:28,222 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-22 14:42:15,194 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17606098 = 2024-04-22 23:18:00 UTC, 97.1% complete
2024-04-22 14:42:59,829 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 363, 56.0% complete
2024-04-22 14:43:35,251 INFO [watch_NEMO_agrif] NEMO AGRIF run 11092903.orca2.ibb on orcinus-nowcast-agrif completed
2024-04-22 14:43:35,256 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-04-22 14:43:35,635 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:43:36,847 INFO [download_results] running in process 2105133
2024-04-22 14:43:36,848 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:43:36,849 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-22 14:43:36,850 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:43:36,852 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/22apr24 to /results/SalishSea/nowcast-agrif.201702
2024-04-22 14:44:00,922 INFO [download_results] nowcast-agrif 2024-04-22 results files from orcinus-nowcast-agrif downloaded
2024-04-22 14:44:00,929 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 14:44:01,108 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:44:04,593 INFO [make_plots] running in process 2105140
2024-04-22 14:44:04,595 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:44:04,596 INFO [make_plots] publishing log messages to tcp://*:5563
2024-04-22 14:44:04,597 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:44:05,438 INFO [make_plots] preparing render list for 2024-04-22 NEMO nowcast-agrif research figures
2024-04-22 14:44:05,440 INFO [make_plots] added baynes_sound_surface figure to 2024-04-22 NEMO nowcast-agrif research render list
2024-04-22 14:44:05,441 INFO [make_plots] starting to render nemo nowcast-agrif research 22apr24 figures
2024-04-22 14:44:14,130 INFO [make_plots] finished rendering nemo nowcast-agrif research 22apr24 figures
2024-04-22 14:44:14,133 INFO [make_plots] nemo research plots for 2024-04-22 nowcast-agrif completed
2024-04-22 14:44:14,137 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-22 14:47:15,197 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17606160 = 2024-04-22 23:59:20 UTC, 100.0% complete
2024-04-22 14:47:59,829 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 441, 68.1% complete
2024-04-22 14:52:15,199 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-04-22 14:52:15,203 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-22 14:52:15,576 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 14:52:16,800 INFO [download_results] running in process 2105372
2024-04-22 14:52:16,801 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 14:52:16,803 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-22 14:52:16,804 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 14:52:16,805 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/22apr24 to /results2/SalishSea/nowcast-green.202111
2024-04-22 14:52:59,892 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 516, 79.6% complete
2024-04-22 14:57:59,891 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 588, 90.7% complete
2024-04-22 15:03:00,382 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-22 15:03:04,347 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240422/18/
2024-04-22 15:03:04,349 INFO [collect_weather] 2024-04-22 2.5km weather forecast 18 collection complete
2024-04-22 15:03:04,351 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-22 15:03:04,738 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-04-22 15:03:04,747 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-04-22 15:03:04,752 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-04-22 15:03:04,757 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-04-23'], host='localhost')
2024-04-22 15:03:05,740 INFO [download_weather] running in process 2105399
2024-04-22 15:03:05,743 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:03:05,745 INFO [download_weather] publishing log messages to tcp://*:5564
2024-04-22 15:03:05,746 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:03:05,747 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240422
2024-04-22 15:03:05,748 INFO [collect_weather] running in process 2105400
2024-04-22 15:03:05,749 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:03:05,750 INFO [collect_weather] publishing log messages to tcp://*:5565
2024-04-22 15:03:05,751 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:03:05,755 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-04-22 15:03:05,756 INFO [download_weather] running in process 2105397
2024-04-22 15:03:05,757 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:03:05,758 INFO [download_weather] publishing log messages to tcp://*:5566
2024-04-22 15:03:05,759 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:03:05,760 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240422
2024-04-22 15:03:06,890 INFO [crop_gribs] running in process 2105401
2024-04-22 15:03:06,891 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:03:06,892 INFO [crop_gribs] publishing log messages to tcp://*:5567
2024-04-22 15:03:06,893 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:03:06,894 INFO [crop_gribs] cropping 2024-04-23 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-04-22 15:03:06,905 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240423/00/
2024-04-22 15:07:57,077 INFO [download_weather] 2024-04-22 1km weather forecast 12 downloads complete
2024-04-22 15:07:57,080 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-22 15:07:59,866 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-22 15:07:59,974 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-22 15:08:00,336 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 15:08:01,456 INFO [download_wwatch3_results] running in process 2105461
2024-04-22 15:08:01,457 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:08:01,458 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2024-04-22 15:08:01,459 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:08:09,123 INFO [download_weather] 2024-04-22 1km weather forecast 00 downloads complete
2024-04-22 15:08:09,126 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-22 15:13:33,719 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240422/18/
2024-04-22 15:13:33,720 INFO [crop_gribs] 2024-04-22 18 GRIBs cropping complete
2024-04-22 15:13:33,723 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-22 15:18:05,115 INFO [download_wwatch3_results] forecast 2024-04-22 results files from arbutus.cloud-nowcast downloaded
2024-04-22 15:18:05,119 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-22 15:18:05,308 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 15:18:06,490 INFO [update_forecast_datasets] running in process 2105509
2024-04-22 15:18:06,492 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:18:06,493 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2024-04-22 15:18:06,494 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:18:06,496 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/22apr24 run
2024-04-22 15:18:06,497 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/22apr24 run
2024-04-22 15:18:06,545 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/22apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-22 15:18:06,547 INFO [update_forecast_datasets] wwatch3 2024-04-22 forecast rolling forecast datasets updated
2024-04-22 15:18:06,550 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-22 15:18:06,729 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-22 15:18:07,730 INFO [ping_erddap] running in process 2105514
2024-04-22 15:18:07,732 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:18:07,734 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-04-22 15:18:07,735 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:18:07,738 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-22 15:18:07,741 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-22 15:18:07,916 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 15:18:12,311 INFO [make_plots] running in process 2105519
2024-04-22 15:18:12,313 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:18:12,314 INFO [make_plots] publishing log messages to tcp://*:5564
2024-04-22 15:18:12,315 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:18:12,316 INFO [make_plots] preparing render list for 2024-04-22 WaveWatch3 forecast publish figures
2024-04-22 15:18:12,318 INFO [make_plots] added Halibut Bank figure to 2024-04-22 WaveWatch3 forecast publish render list
2024-04-22 15:18:12,319 INFO [make_plots] added Sentry Shoal figure to 2024-04-22 WaveWatch3 forecast publish render list
2024-04-22 15:18:12,320 INFO [make_plots] starting to render wwatch3 forecast publish 22apr24 figures
2024-04-22 15:18:12,558 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-22 15:18:21,252 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-22 15:18:27,827 INFO [make_plots] finished rendering wwatch3 forecast publish 22apr24 figures
2024-04-22 15:18:27,828 INFO [make_plots] wwatch3 publish plots for 2024-04-22 forecast completed
2024-04-22 15:18:27,832 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-22 15:29:06,785 INFO [download_results] nowcast-green 2024-04-22 results files from arbutus.cloud-nowcast downloaded
2024-04-22 15:29:06,792 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 15:29:06,975 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 15:29:06,981 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-04-22 15:29:06,985 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 15:29:06,991 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 15:29:06,995 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 15:29:08,176 INFO [ping_erddap] running in process 2105574
2024-04-22 15:29:08,177 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:29:08,179 INFO [ping_erddap] publishing log messages to tcp://*:5563
2024-04-22 15:29:08,180 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:29:08,198 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-04-22 15:29:08,201 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-22 15:29:09,718 INFO [make_averaged_dataset] running in process 2105575
2024-04-22 15:29:09,720 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:29:09,721 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2024-04-22 15:29:09,722 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:29:09,723 INFO [make_averaged_dataset] creating day-averaged dataset for 22-Apr-2024 physics
2024-04-22 15:29:09,725 INFO [make_averaged_dataset] running in process 2105577
2024-04-22 15:29:09,726 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:29:09,728 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2024-04-22 15:29:09,729 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:29:09,730 INFO [make_averaged_dataset] creating day-averaged dataset for 22-Apr-2024 biology
2024-04-22 15:29:09,744 INFO [make_averaged_dataset] running in process 2105576
2024-04-22 15:29:09,746 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:29:09,747 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2024-04-22 15:29:09,748 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:29:09,749 INFO [make_averaged_dataset] creating day-averaged dataset for 22-Apr-2024 chemistry
2024-04-22 15:29:11,827 INFO [make_plots] running in process 2105572
2024-04-22 15:29:11,829 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:29:11,830 INFO [make_plots] publishing log messages to tcp://*:5568
2024-04-22 15:29:11,832 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:29:13,000 INFO [make_plots] preparing render list for 2024-04-22 NEMO nowcast-green research figures
2024-04-22 15:29:14,925 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:15,620 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:16,292 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:16,965 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:17,699 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:18,445 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:19,127 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:19,922 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:20,681 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:21,439 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:22,277 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:22,968 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:23,232 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:23,233 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:23,233 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:23,234 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-04-22 NEMO nowcast-green research render list
2024-04-22 15:29:23,235 INFO [make_plots] starting to render nemo nowcast-green research 22apr24 figures
2024-04-22 15:30:56,703 INFO [make_averaged_dataset] day-averaged dataset for 22-Apr-2024 chemistry created
2024-04-22 15:30:56,706 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-22 15:31:14,515 INFO [make_averaged_dataset] day-averaged dataset for 22-Apr-2024 physics created
2024-04-22 15:31:14,518 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-22 15:31:48,151 INFO [make_averaged_dataset] day-averaged dataset for 22-Apr-2024 biology created
2024-04-22 15:31:48,155 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-22 15:49:52,073 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 15:49:52,443 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-04-06'], host='localhost')
2024-04-22 15:49:52,450 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 15:49:52,454 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 15:49:53,534 INFO [download_results] running in process 2105740
2024-04-22 15:49:53,535 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 15:49:53,536 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-22 15:49:53,537 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 15:49:53,538 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06apr20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 15:49:57,696 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 15:50:49,211 INFO [download_results] hindcast 2020-04-06 results files from optimum-hindcast downloaded
2024-04-22 15:50:49,226 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 15:50:49,411 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-04-06'], host='localhost')
2024-04-22 15:50:50,499 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 16:40:53,320 INFO [make_plots] finished rendering nemo nowcast-green research 22apr24 figures
2024-04-22 16:40:53,329 INFO [make_plots] nemo research plots for 2024-04-22 nowcast-green completed
2024-04-22 16:40:53,386 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-22 17:20:02,414 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 17:20:02,832 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-04-11'], host='localhost')
2024-04-22 17:20:02,840 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 17:20:02,845 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 17:20:04,252 INFO [download_results] running in process 2106417
2024-04-22 17:20:04,253 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 17:20:04,255 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-22 17:20:04,256 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 17:20:04,258 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11apr20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 17:20:08,281 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 17:21:21,194 INFO [download_results] hindcast 2020-04-11 results files from optimum-hindcast downloaded
2024-04-22 17:21:21,210 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 17:21:21,426 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-04-11'], host='localhost')
2024-04-22 17:21:22,943 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 18:45:13,022 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 18:45:13,423 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-04-16'], host='localhost')
2024-04-22 18:45:13,429 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 18:45:13,434 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 18:45:14,462 INFO [download_results] running in process 2107395
2024-04-22 18:45:14,463 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 18:45:14,465 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-22 18:45:14,467 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 18:45:14,468 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16apr20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 18:45:18,538 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 18:47:06,661 INFO [download_results] hindcast 2020-04-16 results files from optimum-hindcast downloaded
2024-04-22 18:47:06,675 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 18:47:06,893 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-04-16'], host='localhost')
2024-04-22 18:47:08,167 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 20:10:23,280 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 20:10:23,687 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-04-21'], host='localhost')
2024-04-22 20:10:23,694 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 20:10:23,700 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 20:10:24,767 INFO [download_results] running in process 2107684
2024-04-22 20:10:24,768 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 20:10:24,770 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-22 20:10:24,771 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 20:10:24,772 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21apr20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 20:10:28,749 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 20:11:54,956 INFO [download_results] hindcast 2020-04-21 results files from optimum-hindcast downloaded
2024-04-22 20:11:54,971 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 20:11:55,184 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-04-21'], host='localhost')
2024-04-22 20:11:56,262 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 21:05:36,419 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240423/00/
2024-04-22 21:05:36,420 INFO [collect_weather] 2024-04-23 2.5km weather forecast 00 collection complete
2024-04-22 21:05:36,423 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-22 21:05:36,849 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-04-22 21:05:36,854 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-04-23'], host='localhost')
2024-04-22 21:05:37,926 INFO [collect_weather] running in process 2107873
2024-04-22 21:05:37,927 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 21:05:37,928 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-04-22 21:05:37,929 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-22 21:05:37,941 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-04-22 21:05:39,711 INFO [crop_gribs] running in process 2107876
2024-04-22 21:05:39,712 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 21:05:39,714 INFO [crop_gribs] publishing log messages to tcp://*:5563
2024-04-22 21:05:39,715 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-22 21:05:39,717 INFO [crop_gribs] cropping 2024-04-23 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-04-22 21:05:39,725 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240423/06/
2024-04-22 21:14:08,455 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240423/00/
2024-04-22 21:14:08,456 INFO [crop_gribs] 2024-04-23 00 GRIBs cropping complete
2024-04-22 21:14:08,459 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-22 21:35:33,476 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 21:35:33,869 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-04-26'], host='localhost')
2024-04-22 21:35:33,875 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 21:35:33,879 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 21:35:34,872 INFO [download_results] running in process 2108025
2024-04-22 21:35:34,873 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 21:35:34,875 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-22 21:35:34,876 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 21:35:34,877 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26apr20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 21:35:39,148 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 21:37:03,351 INFO [download_results] hindcast 2020-04-26 results files from optimum-hindcast downloaded
2024-04-22 21:37:03,366 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 21:37:03,588 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-04-26'], host='localhost')
2024-04-22 21:37:04,732 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 21:37:04,951 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2020-apr', 'robot.graham'], host='localhost')
2024-04-22 21:37:05,866 INFO [archive_tarball] running in process 2108077
2024-04-22 21:37:05,869 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 21:37:05,870 INFO [archive_tarball] publishing log messages to tcp://*:5564
2024-04-22 21:37:05,870 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-22 21:37:05,871 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-apr20.tar from /ocean/sallen/timbucktwo/oxygen/*apr20/
2024-04-22 21:58:19,163 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-apr20.index from /ocean/dlatorne/oxygen-apr20.tar
2024-04-22 21:58:19,277 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-apr20.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-22 22:16:08,314 INFO [archive_tarball] hindcast *apr20 results files archived to robot.graham
2024-04-22 22:16:08,317 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-22 23:00:43,541 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 23:00:43,947 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-05-01'], host='localhost')
2024-04-22 23:00:43,962 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 23:00:43,966 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 23:00:45,023 INFO [download_results] running in process 2108450
2024-04-22 23:00:45,024 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 23:00:45,025 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-22 23:00:45,026 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 23:00:45,027 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01may20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 23:00:49,237 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 23:01:33,819 INFO [download_results] hindcast 2020-05-01 results files from optimum-hindcast downloaded
2024-04-22 23:01:33,833 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 23:01:34,042 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-05-01'], host='localhost')
2024-04-22 23:01:34,881 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-23 00:30:53,720 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-23 00:30:54,121 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-05-06'], host='localhost')
2024-04-23 00:30:54,127 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-23 00:30:54,132 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-23 00:30:55,023 INFO [download_results] running in process 2108904
2024-04-23 00:30:55,024 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 00:30:55,025 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-23 00:30:55,026 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-23 00:30:55,028 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06may20 to /ocean/sallen/timbucktwo/oxygen
2024-04-23 00:30:59,164 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-23 00:31:44,478 INFO [download_results] hindcast 2020-05-06 results files from optimum-hindcast downloaded
2024-04-23 00:31:44,494 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-23 00:31:44,704 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-05-06'], host='localhost')
2024-04-23 00:31:45,594 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-23 01:56:03,839 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-23 01:56:04,239 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-05-11'], host='localhost')
2024-04-23 01:56:04,246 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-23 01:56:04,249 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-23 01:56:05,210 INFO [download_results] running in process 2109817
2024-04-23 01:56:05,211 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 01:56:05,212 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-23 01:56:05,213 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-23 01:56:05,214 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11may20 to /ocean/sallen/timbucktwo/oxygen
2024-04-23 01:56:09,384 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-23 01:56:53,785 INFO [download_results] hindcast 2020-05-11 results files from optimum-hindcast downloaded
2024-04-23 01:56:53,802 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-23 01:56:54,018 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-05-11'], host='localhost')
2024-04-23 01:56:55,071 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-23 03:02:12,603 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240423/06/
2024-04-23 03:02:12,604 INFO [collect_weather] 2024-04-23 2.5km weather forecast 06 collection complete
2024-04-23 03:02:12,607 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-23 03:02:13,008 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,015 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,020 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,025 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,030 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,035 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,041 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,046 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,051 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,057 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,066 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,089 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,101 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,109 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-04-22'], host='localhost')
2024-04-23 03:02:13,114 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-04-23 03:02:13,119 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-04-23 03:02:13,125 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-04-23 03:02:13,139 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-04-23 03:02:13,147 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-04-23 03:02:13,158 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-04-23 03:02:15,186 INFO [collect_NeahBay_ssh] running in process 2110846
2024-04-23 03:02:15,189 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:15,190 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2024-04-23 03:02:15,190 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:15,191 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-04-23
2024-04-23 03:02:15,207 INFO [collect_weather] running in process 2110847
2024-04-23 03:02:15,208 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:15,209 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-04-23 03:02:15,210 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:15,234 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-04-23 03:02:16,512 INFO [collect_river_data] running in process 2110831
2024-04-23 03:02:16,515 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,516 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-04-23 03:02:16,517 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,519 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-04-22
2024-04-23 03:02:16,555 INFO [collect_river_data] running in process 2110832
2024-04-23 03:02:16,556 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,557 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-04-23 03:02:16,558 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,559 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-04-22
2024-04-23 03:02:16,560 INFO [collect_river_data] running in process 2110836
2024-04-23 03:02:16,561 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,562 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-04-23 03:02:16,566 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,567 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-04-22
2024-04-23 03:02:16,568 INFO [collect_river_data] running in process 2110838
2024-04-23 03:02:16,569 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,570 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-04-23 03:02:16,571 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,572 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-04-22
2024-04-23 03:02:16,573 INFO [collect_river_data] running in process 2110830
2024-04-23 03:02:16,574 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,576 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-23 03:02:16,577 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,578 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-04-22
2024-04-23 03:02:16,697 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-04-23 03:02:16,698 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-04-22 completed
2024-04-23 03:02:16,700 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:16,712 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-04-23 03:02:16,713 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-04-22 completed
2024-04-23 03:02:16,715 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-04-23 03:02:16,716 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-04-22 completed
2024-04-23 03:02:16,740 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-04-22 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-04-23 03:02:16,741 INFO [collect_river_data] ECCC Fraser river data collection for 2024-04-22 completed
2024-04-23 03:02:16,785 INFO [collect_river_data] running in process 2110837
2024-04-23 03:02:16,786 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,787 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-04-23 03:02:16,788 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,789 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-04-22
2024-04-23 03:02:16,851 INFO [collect_river_data] running in process 2110839
2024-04-23 03:02:16,852 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,853 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-04-23 03:02:16,854 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,855 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-04-22
2024-04-23 03:02:16,861 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-04-23 03:02:16,862 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-04-22 completed
2024-04-23 03:02:16,894 INFO [collect_river_data] running in process 2110827
2024-04-23 03:02:16,895 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,896 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-04-23 03:02:16,897 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,898 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-04-22
2024-04-23 03:02:16,899 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-04-22 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-04-23 03:02:16,900 INFO [collect_river_data] ECCC Englishman river data collection for 2024-04-22 completed
2024-04-23 03:02:16,903 INFO [collect_river_data] running in process 2110834
2024-04-23 03:02:16,904 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,905 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-04-23 03:02:16,906 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,907 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-04-22
2024-04-23 03:02:16,931 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-04-23 03:02:16,932 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-04-22 completed
2024-04-23 03:02:16,938 INFO [collect_river_data] running in process 2110835
2024-04-23 03:02:16,939 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,940 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-04-23 03:02:16,940 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,941 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-04-22
2024-04-23 03:02:16,951 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:16,986 INFO [collect_river_data] running in process 2110840
2024-04-23 03:02:16,987 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,988 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-04-23 03:02:16,988 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,989 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-04-22
2024-04-23 03:02:16,990 INFO [collect_river_data] running in process 2110833
2024-04-23 03:02:16,991 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,991 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-04-23 03:02:16,992 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,994 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-04-22
2024-04-23 03:02:16,994 INFO [collect_river_data] running in process 2110829
2024-04-23 03:02:16,995 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:16,996 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-04-23 03:02:16,997 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:16,999 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-04-22
2024-04-23 03:02:17,003 INFO [collect_river_data] running in process 2110842
2024-04-23 03:02:17,004 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:17,005 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-04-23 03:02:17,006 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:17,007 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-04-22
2024-04-23 03:02:17,008 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-04-23 03:02:17,010 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-04-22 completed
2024-04-23 03:02:17,024 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-04-23 03:02:17,025 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-04-22 completed
2024-04-23 03:02:17,053 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-04-22 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-04-23 03:02:17,054 INFO [collect_river_data] ECCC Capilano river data collection for 2024-04-22 completed
2024-04-23 03:02:17,062 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-04-23 03:02:17,063 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-04-22 completed
2024-04-23 03:02:17,072 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-04-23 03:02:17,073 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-04-22 completed
2024-04-23 03:02:17,082 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-04-23 03:02:17,083 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-04-22 completed
2024-04-23 03:02:17,098 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-04-22 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-04-23 03:02:17,099 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-04-22 completed
2024-04-23 03:02:17,100 INFO [collect_NeahBay_ssh] 2024-04-23T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-04-23 03:02:17,199 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:17,277 INFO [crop_gribs] running in process 2110848
2024-04-23 03:02:17,278 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:17,278 INFO [crop_gribs] publishing log messages to tcp://*:5590
2024-04-23 03:02:17,279 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:17,280 INFO [crop_gribs] cropping 2024-04-23 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-04-23 03:02:17,287 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240423/12/
2024-04-23 03:02:17,455 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:17,596 INFO [get_onc_ctd] running in process 2110844
2024-04-23 03:02:17,597 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:17,598 INFO [get_onc_ctd] publishing log messages to tcp://*:5568
2024-04-23 03:02:17,599 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:17,600 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-04-22
2024-04-23 03:02:17,603 INFO [get_onc_ctd] running in process 2110843
2024-04-23 03:02:17,604 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:17,605 INFO [get_onc_ctd] publishing log messages to tcp://*:5569
2024-04-23 03:02:17,606 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:17,607 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-04-22
2024-04-23 03:02:17,649 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:17,661 INFO [get_onc_ferry] running in process 2110845
2024-04-23 03:02:17,662 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:17,663 INFO [get_onc_ferry] publishing log messages to tcp://*:5591
2024-04-23 03:02:17,664 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:17,665 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-04-22
2024-04-23 03:02:17,851 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:18,036 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:18,220 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:18,419 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:18,600 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:18,804 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:18,986 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:19,170 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:19,361 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-23 03:02:19,546 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-23 03:02:19,731 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-04-23'], host='localhost')
2024-04-23 03:02:22,560 INFO [make_ssh_files] running in process 2111255
2024-04-23 03:02:22,561 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:22,562 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2024-04-23 03:02:22,563 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:22,564 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-04-23 NOAA Neah Bay 00Z observation and forecast values
2024-04-23 03:02:27,309 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d18.nc
2024-04-23 03:02:27,477 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d19.nc
2024-04-23 03:02:27,654 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d20.nc
2024-04-23 03:02:27,844 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d21.nc
2024-04-23 03:02:28,015 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d22.nc
2024-04-23 03:02:28,253 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d23.nc
2024-04-23 03:02:28,425 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d24.nc
2024-04-23 03:02:28,594 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d25.nc
2024-04-23 03:02:28,763 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d26.nc
2024-04-23 03:02:28,901 INFO [make_ssh_files] sea surface height boundary file for 2024-04-23 forecast2 run created
2024-04-23 03:02:28,905 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-23 03:02:29,101 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-23 03:02:29,107 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-23 03:02:30,210 INFO [make_runoff_file] running in process 2111289
2024-04-23 03:02:30,211 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:30,212 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2024-04-23 03:02:30,213 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:30,240 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-23 03:02:30,316 INFO [make_runoff_file] 2024-04-23 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-23 03:02:30,318 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-23 03:02:30,890 INFO [make_v202111_runoff_file] running in process 2111287
2024-04-23 03:02:30,891 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:30,893 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5562
2024-04-23 03:02:30,894 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:30,895 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-22
2024-04-23 03:02:31,922 ERROR [make_v202111_runoff_file] no 2024-04-22 discharge obs for Skagit_MountVernon: patching
2024-04-23 03:02:32,423 ERROR [make_v202111_runoff_file] no 2024-04-22 discharge obs for Snohomish_Monroe: patching
2024-04-23 03:02:32,674 ERROR [make_v202111_runoff_file] no 2024-04-22 discharge obs for Nisqually_McKenna: patching
2024-04-23 03:02:32,794 ERROR [make_v202111_runoff_file] no 2024-04-22 discharge obs for Greenwater_Greenwater: patching
2024-04-23 03:02:34,037 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-22: /results/forcing/rivers/R202108Dailies_y2024m04d22.nc
2024-04-23 03:02:34,039 INFO [make_v202111_runoff_file] 2024-04-22 runoff file creation completed
2024-04-23 03:02:34,041 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-23 03:02:36,291 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-04-22
2024-04-23 03:02:37,152 INFO [get_onc_ctd] 2024-04-22 ONC SEVIP CTD T&S file created
2024-04-23 03:02:37,154 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-04-23 03:02:37,336 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-04-23 03:02:38,164 INFO [ping_erddap] running in process 2111329
2024-04-23 03:02:38,165 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:02:38,166 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-23 03:02:38,167 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:02:38,168 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-04-23 03:02:38,170 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-23 03:02:53,310 CRITICAL [get_onc_ctd] 2024-04-22 ONC SCVIP CTD T&S file creation failed
2024-04-23 03:03:01,657 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-04-22
2024-04-23 03:03:29,254 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-04-22
2024-04-23 03:04:00,686 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-04-22
2024-04-23 03:04:16,829 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-04-22
2024-04-23 03:04:33,680 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-04-22
2024-04-23 03:04:42,386 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-04-22
2024-04-23 03:04:51,180 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-04-22; substituting empty dataset
2024-04-23 03:04:51,181 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-04-22
2024-04-23 03:04:59,408 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-22; substituting empty dataset
2024-04-23 03:06:52,075 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-04-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-04-23 03:06:52,093 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-04-23 03:06:52,419 INFO [get_onc_ferry] 2024-04-22 ONC TWDP ferry data file created
2024-04-23 03:06:52,422 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-04-23 03:06:52,645 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-04-23 03:06:53,494 INFO [ping_erddap] running in process 2111360
2024-04-23 03:06:53,495 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:06:53,497 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-23 03:06:53,498 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:06:53,500 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-04-23 03:06:53,502 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-23 03:11:02,959 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240423/06/
2024-04-23 03:11:02,960 INFO [crop_gribs] 2024-04-23 06 GRIBs cropping complete
2024-04-23 03:11:02,964 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-23 03:11:03,386 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-04-23 03:11:05,078 INFO [grib_to_netcdf] running in process 2111379
2024-04-23 03:11:05,079 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:11:05,080 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2024-04-23 03:11:05,082 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:11:05,083 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-23 forecast2 run
2024-04-23 03:11:16,555 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d24.nc
2024-04-23 03:11:20,099 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d25.nc
2024-04-23 03:11:20,105 INFO [grib_to_netcdf] 2024-04-23 NEMO-atmos forcing file for forecast2 created
2024-04-23 03:11:20,108 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-23 03:11:20,331 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-04-23 03:11:20,335 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-04-23 03:11:20,339 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-04-23 03:11:20,344 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-04-23 03:11:21,219 INFO [upload_forcing] running in process 2111415
2024-04-23 03:11:21,220 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:11:21,222 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-04-23 03:11:21,223 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:11:21,231 INFO [upload_forcing] running in process 2111418
2024-04-23 03:11:21,232 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:11:21,233 INFO [upload_forcing] publishing log messages to tcp://*:5563
2024-04-23 03:11:21,234 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:11:21,253 INFO [upload_forcing] running in process 2111417
2024-04-23 03:11:21,254 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:11:21,256 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-23 03:11:21,257 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:11:21,259 INFO [upload_forcing] running in process 2111419
2024-04-23 03:11:21,260 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:11:21,260 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-23 03:11:21,261 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:11:22,759 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m04d22.nc
2024-04-23 03:11:22,761 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m04d22.nc
2024-04-23 03:11:22,837 INFO [upload_forcing] forecast2 2024-04-23 forcing files upload to optimum-hindcast completed
2024-04-23 03:11:22,839 INFO [upload_forcing] forecast2 2024-04-23 forcing files upload to orcinus-nowcast-agrif completed
2024-04-23 03:11:22,841 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-23 03:11:23,054 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-23 03:11:24,745 INFO [upload_forcing] forecast2 2024-04-23 forcing files upload to arbutus.cloud-nowcast completed
2024-04-23 03:11:24,751 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-23 03:11:24,972 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-23'], host='localhost')
2024-04-23 03:11:26,029 INFO [make_forcing_links] running in process 2111447
2024-04-23 03:11:26,030 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:11:26,032 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-04-23 03:11:26,033 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:11:27,234 INFO [make_forcing_links] forecast2 2024-04-23 forcing file links on arbutus.cloud-nowcast created
2024-04-23 03:11:27,238 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-23 03:11:27,453 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-23'], host='arbutus.cloud-nowcast')
2024-04-23 03:11:29,573 INFO [run_NEMO] running in process 20023
2024-04-23 03:11:29,576 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:11:29,578 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-23 03:11:29,701 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:11:30,807 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/22apr24forecast2_2024-04-23T031129.785192-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-23 03:11:30,829 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-23 03:11:30,847 INFO [run_NEMO] forecast2 NEMO run for 2024-04-23 on arbutus.cloud-nowcast started
2024-04-23 03:11:30,855 INFO [upload_forcing] forecast2 2024-04-23 forcing files upload to robot.graham completed
2024-04-23 03:11:31,053 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-23 03:11:31,059 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-23 03:11:32,938 INFO [watch_NEMO] running in process 20276
2024-04-23 03:11:32,939 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:11:32,940 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-23 03:11:32,942 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:11:33,018 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-23 03:16:33,021 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17608761 = 2024-04-24 04:53:20 UTC, 16.3% complete
2024-04-23 03:21:13,818 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-23 03:21:14,234 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-05-16'], host='localhost')
2024-04-23 03:21:14,241 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-23 03:21:14,245 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-23 03:21:15,201 INFO [download_results] running in process 2111621
2024-04-23 03:21:15,202 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:21:15,203 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-23 03:21:15,204 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:21:15,206 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16may20 to /ocean/sallen/timbucktwo/oxygen
2024-04-23 03:21:19,364 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-23 03:21:33,024 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17609244 = 2024-04-24 10:15:20 UTC, 34.2% complete
2024-04-23 03:22:03,734 INFO [download_results] hindcast 2020-05-16 results files from optimum-hindcast downloaded
2024-04-23 03:22:03,748 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-23 03:22:03,961 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-05-16'], host='localhost')
2024-04-23 03:22:04,940 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-23 03:26:33,026 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17609727 = 2024-04-24 15:37:20 UTC, 52.1% complete
2024-04-23 03:31:33,028 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17610224 = 2024-04-24 21:08:40 UTC, 70.5% complete
2024-04-23 03:36:33,029 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17610688 = 2024-04-25 02:18:00 UTC, 87.7% complete
2024-04-23 03:41:33,031 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17611020 = 2024-04-25 05:59:20 UTC, 100.0% complete
2024-04-23 03:46:33,036 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-04-23 03:46:33,040 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-23 03:46:33,450 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-23'], host='arbutus.cloud-nowcast')
2024-04-23 03:46:33,457 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-23'], host='arbutus.cloud-nowcast')
2024-04-23 03:46:33,461 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-23 03:46:34,643 INFO [download_results] running in process 2111787
2024-04-23 03:46:34,644 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 03:46:34,646 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-23 03:46:34,647 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-23 03:46:34,648 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/22apr24 to /results/SalishSea/forecast2.202111
2024-04-23 03:46:39,914 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-23 03:48:27,809 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-23 03:48:28,037 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-23'], host='arbutus.cloud-nowcast')
2024-04-23 03:48:36,264 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-23 03:48:36,489 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-23 03:48:50,056 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-23 03:53:49,978 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 75, 13.9% complete
2024-04-23 03:58:50,281 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 156, 28.9% complete
2024-04-23 04:03:50,219 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 237, 43.9% complete
2024-04-23 04:08:50,178 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 315, 58.3% complete
2024-04-23 04:11:17,091 INFO [download_results] forecast2 2024-04-22 results files from arbutus.cloud-nowcast downloaded
2024-04-23 04:11:17,095 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-23 04:11:17,306 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-23 04:11:19,168 INFO [make_CHS_currents_file] running in process 2112086
2024-04-23 04:11:19,169 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:11:19,170 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2024-04-23 04:11:19,172 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:11:53,701 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-22 for forecast2
2024-04-23 04:11:53,704 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-23 04:11:53,919 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-23 04:11:54,824 INFO [update_forecast_datasets] running in process 2112147
2024-04-23 04:11:54,825 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:11:54,827 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2024-04-23 04:11:54,828 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:11:54,829 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/22apr24 run
2024-04-23 04:13:49,642 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 390, 72.2% complete
2024-04-23 04:16:08,179 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/22apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-23 04:16:08,180 INFO [update_forecast_datasets] nemo 2024-04-22 forecast2 rolling forecast datasets updated
2024-04-23 04:16:08,183 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-23 04:16:08,406 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-23 04:16:08,411 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-04-22'], host='localhost')
2024-04-23 04:16:08,415 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-23 04:16:09,410 INFO [ping_erddap] running in process 2112222
2024-04-23 04:16:09,412 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:16:09,413 INFO [ping_erddap] publishing log messages to tcp://*:5563
2024-04-23 04:16:09,414 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:16:09,424 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-23 04:16:09,426 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-23 04:16:10,645 INFO [make_surface_current_tiles] running in process 2112225
2024-04-23 04:16:10,646 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:16:10,647 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2024-04-23 04:16:10,647 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:16:10,650 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-22 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/22apr24
2024-04-23 04:16:12,592 INFO [make_plots] running in process 2112224
2024-04-23 04:16:12,593 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:16:12,593 INFO [make_plots] publishing log messages to tcp://*:5565
2024-04-23 04:16:12,594 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:16:13,347 INFO [make_plots] preparing render list for 2024-04-22 NEMO forecast2 publish figures
2024-04-23 04:16:13,442 INFO [make_plots] added Website_thumbnail figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,443 INFO [make_plots] added Threshold_website figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,444 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,445 INFO [make_plots] added Boundary Bay figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,445 INFO [make_plots] added Campbell River figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,446 INFO [make_plots] added Cherry Point figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,447 INFO [make_plots] added Friday Harbor figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,448 INFO [make_plots] added Halfmoon Bay figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,448 INFO [make_plots] added Nanaimo figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,449 INFO [make_plots] added Neah Bay figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,450 INFO [make_plots] added New Westminster figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,451 INFO [make_plots] added Patricia Bay figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,452 INFO [make_plots] added Point Atkinson figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,452 INFO [make_plots] added Port Renfrew figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,453 INFO [make_plots] added Sand Heads figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,454 INFO [make_plots] added Sandy Cove figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,454 INFO [make_plots] added Squamish figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,455 INFO [make_plots] added Victoria figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,456 INFO [make_plots] added Woodwards Landing figure to 2024-04-22 NEMO forecast2 publish render list
2024-04-23 04:16:13,457 INFO [make_plots] starting to render nemo forecast2 publish 22apr24 figures
2024-04-23 04:18:49,641 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 471, 87.2% complete
2024-04-23 04:23:49,634 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-23 04:28:49,635 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-23 04:28:49,638 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-23 04:28:50,334 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-23'], host='localhost')
2024-04-23 04:28:51,835 INFO [download_wwatch3_results] running in process 2114159
2024-04-23 04:28:51,836 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:28:51,837 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-04-23 04:28:51,838 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:30:52,579 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/22apr24
2024-04-23 04:31:08,181 INFO [download_wwatch3_results] forecast2 2024-04-23 results files from arbutus.cloud-nowcast downloaded
2024-04-23 04:31:08,184 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-23 04:31:08,398 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-04-23'], host='localhost')
2024-04-23 04:31:09,353 INFO [update_forecast_datasets] running in process 2114413
2024-04-23 04:31:09,355 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:31:09,356 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566
2024-04-23 04:31:09,357 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:31:09,358 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/23apr24 run
2024-04-23 04:31:09,359 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/23apr24 run
2024-04-23 04:31:34,085 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/23apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-23 04:31:34,086 INFO [update_forecast_datasets] wwatch3 2024-04-23 forecast2 rolling forecast datasets updated
2024-04-23 04:31:34,089 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-23 04:31:34,312 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-23 04:31:35,150 INFO [ping_erddap] running in process 2114426
2024-04-23 04:31:35,151 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:31:35,152 INFO [ping_erddap] publishing log messages to tcp://*:5566
2024-04-23 04:31:35,153 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:31:35,154 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-23 04:31:35,156 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-23 04:31:35,379 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-04-23'], host='localhost')
2024-04-23 04:31:38,868 INFO [make_plots] running in process 2114431
2024-04-23 04:31:38,869 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:31:38,870 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-23 04:31:38,871 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:31:38,872 INFO [make_plots] preparing render list for 2024-04-23 WaveWatch3 forecast2 publish figures
2024-04-23 04:31:38,873 INFO [make_plots] added Halibut Bank figure to 2024-04-23 WaveWatch3 forecast2 publish render list
2024-04-23 04:31:38,874 INFO [make_plots] added Sentry Shoal figure to 2024-04-23 WaveWatch3 forecast2 publish render list
2024-04-23 04:31:38,875 INFO [make_plots] starting to render wwatch3 forecast2 publish 23apr24 figures
2024-04-23 04:31:39,084 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-23 04:31:45,722 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-23 04:31:51,523 INFO [make_plots] finished rendering wwatch3 forecast2 publish 23apr24 figures
2024-04-23 04:31:51,524 INFO [make_plots] wwatch3 publish plots for 2024-04-23 forecast2 completed
2024-04-23 04:31:51,528 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-23 04:34:28,930 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-23 04:34:28,931 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-23 04:34:32,854 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/22apr24
2024-04-23 04:34:32,862 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-22 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/22apr24
2024-04-23 04:34:32,863 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-22 forecast2 completed
2024-04-23 04:34:33,139 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-23 04:35:04,048 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:36:52,700 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:36:53,122 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:37:28,480 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:38:39,726 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:39:15,535 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:39:51,892 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:40:28,535 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:41:04,525 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:41:40,551 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:42:17,121 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:42:17,538 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:42:53,505 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:43:29,025 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-18 00:00:00Z to 2024-04-25 05:55:00Z
2024-04-23 04:44:04,783 INFO [make_plots] finished rendering nemo forecast2 publish 22apr24 figures
2024-04-23 04:44:04,785 INFO [make_plots] nemo publish plots for 2024-04-22 forecast2 completed
2024-04-23 04:44:04,793 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-23 04:44:05,117 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-23 04:44:08,052 INFO [make_feeds] running in process 2114586
2024-04-23 04:44:08,056 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:44:08,057 INFO [make_feeds] publishing log messages to tcp://*:5563
2024-04-23 04:44:08,058 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:44:35,945 INFO [make_feeds] ATOM feeds for 2024-04-22 forecast2 run completed
2024-04-23 04:44:35,949 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-23 04:44:36,280 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-04-23 04:44:37,150 INFO [clear_checklist] running in process 2114612
2024-04-23 04:44:37,151 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:44:37,152 INFO [clear_checklist] publishing log messages to tcp://*:5563
2024-04-23 04:44:37,153 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:44:37,154 INFO [clear_checklist] requesting that manager clear system state checklist
2024-04-23 04:44:37,155 INFO [manager] writing checklist to log file
2024-04-23 04:44:37,206 INFO [manager] checklist cleared
2024-04-23 04:44:37,209 INFO [clear_checklist] nowcast system checklist cleared
2024-04-23 04:44:37,211 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-04-23 04:44:38,039 INFO [rotate_logs] running in process 2114617
2024-04-23 04:44:38,041 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-23 04:44:38,042 INFO [rotate_logs] publishing log messages to tcp://*:5563
2024-04-23 04:44:38,043 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-04-23 04:44:38,044 INFO [rotate_logs] rotating log files