2024-07-23 04:27:04,630 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-07-23 04:27:04,636 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-07-23 04:27:04,642 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-07-23 04:27:04,643 INFO [rotate_logs] log files rotated
2024-07-23 04:27:04,649 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-07-23 04:27:21,870 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 04:27:22,023 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-08-16'], host='localhost')
2024-07-23 04:27:22,029 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 04:27:22,036 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 04:27:23,199 INFO [download_results] running in process 740220
2024-07-23 04:27:23,200 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:27:23,201 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-23 04:27:23,203 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:27:23,204 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16aug18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 04:27:27,342 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 04:28:07,976 INFO [download_results] hindcast 2018-08-16 results files from optimum-hindcast downloaded
2024-07-23 04:28:07,986 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 04:28:07,992 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-08-16'], host='localhost')
2024-07-23 04:28:09,552 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 05:52:31,896 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 05:52:32,064 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-08-21'], host='localhost')
2024-07-23 05:52:32,071 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 05:52:32,076 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 05:52:33,304 INFO [download_results] running in process 740473
2024-07-23 05:52:33,305 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 05:52:33,306 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-23 05:52:33,307 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 05:52:33,308 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21aug18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 05:52:37,570 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 05:53:13,157 INFO [download_results] hindcast 2018-08-21 results files from optimum-hindcast downloaded
2024-07-23 05:53:13,167 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 05:53:13,173 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-08-21'], host='localhost')
2024-07-23 05:53:14,367 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 07:32:43,699 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 07:32:43,867 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-08-26'], host='localhost')
2024-07-23 07:32:43,874 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 07:32:43,880 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 07:32:45,118 INFO [download_results] running in process 741965
2024-07-23 07:32:45,119 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 07:32:45,121 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-23 07:32:45,122 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 07:32:45,123 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26aug18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 07:32:49,329 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 07:33:30,065 INFO [download_results] hindcast 2018-08-26 results files from optimum-hindcast downloaded
2024-07-23 07:33:30,077 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 07:33:30,084 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-08-26'], host='localhost')
2024-07-23 07:33:31,359 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 07:33:31,368 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-aug', 'robot.graham'], host='localhost')
2024-07-23 07:33:32,464 INFO [archive_tarball] running in process 742010
2024-07-23 07:33:32,466 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 07:33:32,467 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-07-23 07:33:32,468 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-23 07:33:32,469 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-aug18.tar from /ocean/sallen/timbucktwo/oxygen/*aug18/
2024-07-23 07:39:30,247 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-aug18.index from /ocean/dlatorne/oxygen-aug18.tar
2024-07-23 07:39:30,334 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-aug18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-23 07:55:34,647 INFO [archive_tarball] hindcast *aug18 results files archived to robot.graham
2024-07-23 07:55:34,651 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-23 09:02:54,143 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 09:02:54,544 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-01'], host='localhost')
2024-07-23 09:02:54,555 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 09:02:54,560 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 09:02:56,434 INFO [download_results] running in process 742661
2024-07-23 09:02:56,435 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:02:56,437 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-23 09:02:56,438 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:02:56,439 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01sep18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 09:03:00,706 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 09:03:38,555 INFO [download_results] hindcast 2018-09-01 results files from optimum-hindcast downloaded
2024-07-23 09:03:38,566 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 09:03:38,575 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-01'], host='localhost')
2024-07-23 09:03:39,762 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 09:11:36,100 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240723/12/
2024-07-23 09:11:36,101 INFO [collect_weather] 2024-07-23 2.5km weather forecast 12 collection complete
2024-07-23 09:11:36,105 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-23 09:11:36,285 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 09:11:36,291 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 09:11:36,295 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 09:11:36,300 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 09:11:36,303 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-07-23 09:11:36,312 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-07-23 09:11:36,315 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-07-23 09:11:36,319 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-07-23 09:11:36,323 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-07-23 09:11:37,962 INFO [collect_weather] running in process 742736
2024-07-23 09:11:37,963 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:37,965 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-07-23 09:11:37,966 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:37,973 INFO [collect_NeahBay_ssh] running in process 742733
2024-07-23 09:11:37,974 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:37,975 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5565
2024-07-23 09:11:37,976 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:37,976 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-07-23
2024-07-23 09:11:37,986 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-07-23 09:11:38,312 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240723/12/
2024-07-23 09:11:38,362 INFO [crop_gribs] 2024-07-23 12 GRIBs cropping complete
2024-07-23 09:11:38,364 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-23 09:11:38,531 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-07-23 09:11:38,700 INFO [collect_river_data] running in process 742727
2024-07-23 09:11:38,701 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:38,702 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-07-23 09:11:38,703 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:38,704 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-07-22
2024-07-23 09:11:38,734 INFO [collect_river_data] running in process 742731
2024-07-23 09:11:38,736 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:38,736 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-23 09:11:38,737 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:38,737 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-07-22
2024-07-23 09:11:38,755 INFO [collect_river_data] running in process 742730
2024-07-23 09:11:38,756 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:38,757 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-23 09:11:38,758 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:38,759 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-07-22
2024-07-23 09:11:38,761 INFO [collect_river_data] running in process 742729
2024-07-23 09:11:38,761 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:38,762 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-23 09:11:38,763 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:38,763 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-07-22
2024-07-23 09:11:38,775 INFO [download_live_ocean] running in process 742734
2024-07-23 09:11:38,776 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:38,777 INFO [download_live_ocean] publishing log messages to tcp://*:5570
2024-07-23 09:11:38,777 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:38,778 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-07-23
2024-07-23 09:11:38,889 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-07-22&endDT=2024-07-22 "HTTP/1.1 200 OK"
2024-07-23 09:11:38,894 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-07-23 09:11:38,896 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-07-22 completed
2024-07-23 09:11:38,899 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 09:11:38,917 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-07-22&endDT=2024-07-22 "HTTP/1.1 200 OK"
2024-07-23 09:11:38,932 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-07-23 09:11:38,933 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-07-22 completed
2024-07-23 09:11:38,935 INFO [make_turbidity_file] running in process 742732
2024-07-23 09:11:38,936 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 09:11:38,937 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:38,938 INFO [make_turbidity_file] publishing log messages to tcp://*:5571
2024-07-23 09:11:38,940 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:38,941 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-07-23
2024-07-23 09:11:39,068 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-07-22&endDT=2024-07-22 "HTTP/1.1 200 OK"
2024-07-23 09:11:39,074 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-07-23 09:11:39,075 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-07-22 completed
2024-07-23 09:11:39,077 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 09:11:39,092 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-07-22&endDT=2024-07-22 "HTTP/1.1 200 OK"
2024-07-23 09:11:39,111 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-07-23 09:11:39,112 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-07-22 completed
2024-07-23 09:11:39,116 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 09:11:39,193 INFO [collect_NeahBay_ssh] 2024-07-23T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-07-23 09:11:39,195 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-23 09:11:39,204 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 09:11:39,305 INFO [crop_gribs] running in process 742737
2024-07-23 09:11:39,307 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:39,307 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-07-23 09:11:39,308 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:39,309 INFO [crop_gribs] cropping 2024-07-23 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-07-23 09:11:39,324 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240723/18/
2024-07-23 09:11:39,632 INFO [download_live_ocean] downloaded 10116606 bytes from https://liveocean.apl.uw.edu/output/f2024.07.23/ubc.nc to /results/forcing/LiveOcean/downloaded/20240723/low_passed_UBC.nc
2024-07-23 09:11:39,633 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-07-23 09:11:40,558 INFO [grib_to_netcdf] running in process 742907
2024-07-23 09:11:40,561 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:40,562 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565
2024-07-23 09:11:40,563 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:40,565 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-23 nowcast and forecast runs
2024-07-23 09:11:40,677 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240723/low_passed_UBC.nc
2024-07-23 09:11:40,677 INFO [download_live_ocean] 2024-07-23 Live Ocean file for Salish Sea western boundary downloaded
2024-07-23 09:11:40,680 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-07-23 09:11:40,841 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-07-23'], host='localhost')
2024-07-23 09:11:41,863 INFO [make_ssh_files] running in process 742929
2024-07-23 09:11:41,864 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:41,865 INFO [make_ssh_files] publishing log messages to tcp://*:5567
2024-07-23 09:11:41,866 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:41,868 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-07-23 NOAA Neah Bay 06Z observation and forecast values
2024-07-23 09:11:41,946 WARNING [make_turbidity_file] Insufficient data after interpolation to create Fraser River turbidity file for 2024-07-23
2024-07-23 09:11:41,947 INFO [make_turbidity_file] 2024-07-23 Fraser River turbidity file creation complete
2024-07-23 09:11:42,892 INFO [make_live_ocean_files] running in process 742963
2024-07-23 09:11:42,893 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:42,894 INFO [make_live_ocean_files] publishing log messages to tcp://*:5568
2024-07-23 09:11:42,895 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:42,897 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-07-23 Live Ocean run
2024-07-23 09:11:46,200 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc
2024-07-23 09:11:46,368 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-23 09:11:46,534 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-23 09:11:46,710 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d22.nc
2024-07-23 09:11:46,921 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d23.nc
2024-07-23 09:11:47,133 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d24.nc
2024-07-23 09:11:47,321 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d25.nc
2024-07-23 09:11:47,508 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d26.nc
2024-07-23 09:11:47,619 INFO [make_ssh_files] sea surface height boundary file for 2024-07-23 nowcast run created
2024-07-23 09:11:47,623 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-23 09:11:47,636 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-23 09:11:47,641 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-23 09:11:48,990 INFO [make_runoff_file] running in process 743012
2024-07-23 09:11:48,991 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:48,992 INFO [make_runoff_file] publishing log messages to tcp://*:5567
2024-07-23 09:11:48,993 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:49,017 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-23 09:11:49,069 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m07d23.nc
2024-07-23 09:11:49,096 INFO [make_runoff_file] 2024-07-23 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-23 09:11:49,099 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-23 09:11:49,536 INFO [make_v202111_runoff_file] running in process 743010
2024-07-23 09:11:49,537 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:11:49,538 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5567
2024-07-23 09:11:49,539 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:11:49,540 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-22
2024-07-23 09:11:52,161 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-22: /results/forcing/rivers/R202108Dailies_y2024m07d22.nc
2024-07-23 09:11:52,162 INFO [make_v202111_runoff_file] 2024-07-22 runoff file creation completed
2024-07-23 09:11:52,164 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-23 09:11:56,073 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d24.nc
2024-07-23 09:11:59,908 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d25.nc
2024-07-23 09:11:59,921 INFO [grib_to_netcdf] 2024-07-23 NEMO-atmos forcing file for nowcast+ created
2024-07-23 09:11:59,924 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-23 09:12:23,077 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d23.nc
2024-07-23 09:12:23,078 INFO [make_live_ocean_files] 2024-07-23 Live Ocean western boundary conditions files created
2024-07-23 09:12:23,080 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-07-23 09:12:23,094 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-07-23 09:12:23,098 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-07-23 09:12:23,103 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-07-23 09:12:23,108 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-07-23 09:12:23,114 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-07-23 09:12:24,283 INFO [ping_erddap] running in process 743052
2024-07-23 09:12:24,285 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:12:24,286 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-23 09:12:24,287 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:12:24,289 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-07-23 09:12:24,291 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 09:12:24,376 INFO [upload_forcing] running in process 743057
2024-07-23 09:12:24,377 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:12:24,379 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-23 09:12:24,380 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:12:24,387 INFO [upload_forcing] running in process 743054
2024-07-23 09:12:24,388 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:12:24,390 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-23 09:12:24,391 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:12:24,397 INFO [upload_forcing] running in process 743055
2024-07-23 09:12:24,399 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:12:24,400 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-23 09:12:24,401 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:12:24,421 INFO [upload_forcing] running in process 743056
2024-07-23 09:12:24,422 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:12:24,423 INFO [upload_forcing] publishing log messages to tcp://*:5570
2024-07-23 09:12:24,425 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:12:26,373 INFO [upload_forcing] nowcast+ 2024-07-23 forcing files upload to optimum-hindcast completed
2024-07-23 09:12:26,378 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 09:12:26,809 INFO [upload_forcing] nowcast+ 2024-07-23 forcing files upload to orcinus-nowcast-agrif completed
2024-07-23 09:12:26,814 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 09:12:35,182 INFO [upload_forcing] nowcast+ 2024-07-23 forcing files upload to robot.graham completed
2024-07-23 09:12:35,186 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 09:12:37,106 INFO [upload_forcing] nowcast+ 2024-07-23 forcing files upload to arbutus.cloud-nowcast completed
2024-07-23 09:12:37,111 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 09:12:37,127 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 09:12:38,673 INFO [make_forcing_links] running in process 743092
2024-07-23 09:12:38,674 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:12:38,675 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-23 09:12:38,676 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:12:39,848 INFO [make_forcing_links] nowcast+ 2024-07-23 forcing file links on arbutus.cloud-nowcast created
2024-07-23 09:12:39,853 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-23 09:12:39,869 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 09:12:41,986 INFO [run_NEMO] running in process 30045
2024-07-23 09:12:41,987 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:12:41,988 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-23 09:12:42,061 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:12:42,896 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/23jul24nowcast_2024-07-23T091242.036974-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-23 09:12:42,910 INFO [run_NEMO] nowcast NEMO run for 2024-07-23 on arbutus.cloud-nowcast started
2024-07-23 09:12:42,920 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-23 09:12:42,924 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-23 09:12:45,070 INFO [watch_NEMO] running in process 30297
2024-07-23 09:12:45,071 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:12:45,073 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-23 09:12:45,076 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:12:45,149 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-23 09:17:45,151 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17803145 = 2024-07-23 04:42:40 UTC, 19.6% complete
2024-07-23 09:22:45,154 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17803646 = 2024-07-23 10:16:40 UTC, 42.8% complete
2024-07-23 09:27:45,158 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17804111 = 2024-07-23 15:26:40 UTC, 64.4% complete
2024-07-23 09:32:45,157 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17804576 = 2024-07-23 20:36:40 UTC, 85.9% complete
2024-07-23 09:37:45,161 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-07-23 09:37:45,164 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-23 09:37:45,329 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 09:37:45,337 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 09:37:46,994 INFO [make_forcing_links] running in process 743162
2024-07-23 09:37:46,996 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:37:46,997 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-23 09:37:46,999 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:37:47,002 INFO [download_results] running in process 743163
2024-07-23 09:37:47,004 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:37:47,005 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-23 09:37:47,007 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:37:47,008 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/23jul24 to /results/SalishSea/nowcast-blue.202111
2024-07-23 09:37:48,050 INFO [make_forcing_links] ssh 2024-07-23 forcing file links on arbutus.cloud-nowcast created
2024-07-23 09:37:48,054 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-23 09:37:48,068 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 09:37:50,250 INFO [run_NEMO] running in process 30671
2024-07-23 09:37:50,252 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:37:50,253 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-23 09:37:50,255 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:37:51,177 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/23jul24forecast_2024-07-23T093750.335585-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-23 09:37:51,197 INFO [run_NEMO] forecast NEMO run for 2024-07-23 on arbutus.cloud-nowcast started
2024-07-23 09:37:51,201 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-23 09:37:51,216 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-23 09:37:53,577 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-23 09:42:53,600 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17805305 = 2024-07-24 04:42:40 UTC, 13.1% complete
2024-07-23 09:47:53,579 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17805779 = 2024-07-24 09:58:40 UTC, 27.7% complete
2024-07-23 09:52:53,571 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17806230 = 2024-07-24 14:59:20 UTC, 41.6% complete
2024-07-23 09:57:53,729 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17806694 = 2024-07-24 20:08:40 UTC, 56.0% complete
2024-07-23 09:58:03,672 INFO [download_results] nowcast 2024-07-23 results files from arbutus.cloud-nowcast downloaded
2024-07-23 09:58:03,676 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 09:58:03,692 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 09:58:03,700 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-07-22'], host='localhost')
2024-07-23 09:58:03,705 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 09:58:06,986 INFO [make_CHS_currents_file] running in process 743281
2024-07-23 09:58:06,987 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:58:06,989 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-07-23 09:58:06,993 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:58:09,364 INFO [make_plots] running in process 743278
2024-07-23 09:58:09,365 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:58:09,367 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-23 09:58:09,371 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:58:09,382 INFO [make_plots] running in process 743280
2024-07-23 09:58:09,385 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 09:58:09,386 INFO [make_plots] publishing log messages to tcp://*:5568
2024-07-23 09:58:09,387 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-23 09:58:10,910 INFO [make_plots] preparing render list for 2024-07-23 NEMO nowcast-blue research figures
2024-07-23 09:58:10,911 INFO [make_plots] preparing render list for 2024-07-22 NEMO nowcast-blue comparison figures
2024-07-23 09:58:11,818 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-blue research render list
2024-07-23 09:58:12,504 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-blue research render list
2024-07-23 09:58:12,505 INFO [make_plots] added Currents_sections_and_surface figure to 2024-07-23 NEMO nowcast-blue research render list
2024-07-23 09:58:12,506 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-07-23 NEMO nowcast-blue research render list
2024-07-23 09:58:12,507 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-07-23 NEMO nowcast-blue research render list
2024-07-23 09:58:12,509 INFO [make_plots] starting to render nemo nowcast research 23jul24 figures
2024-07-23 09:58:36,701 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-23 for nowcast
2024-07-23 09:58:36,704 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-23 09:59:00,549 INFO [make_plots] added SH_wind figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,550 INFO [make_plots] added Compare_VENUS_East figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,551 INFO [make_plots] added Compare_VENUS_Central figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,553 INFO [make_plots] added Central_ADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,554 INFO [make_plots] added Central_depavADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,555 INFO [make_plots] added Central_timeavADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,557 INFO [make_plots] added East_ADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,558 INFO [make_plots] added East_depavADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,559 INFO [make_plots] added East_timeavADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,561 INFO [make_plots] added ddl_ADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,563 INFO [make_plots] added ddl_depavADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,564 INFO [make_plots] added ddl_timeavADCP figure to 2024-07-22 NEMO nowcast-blue comparison render list
2024-07-23 09:59:00,566 INFO [make_plots] starting to render nemo nowcast comparison 22jul24 figures
2024-07-23 09:59:22,481 INFO [make_plots] finished rendering nemo nowcast research 23jul24 figures
2024-07-23 09:59:22,484 INFO [make_plots] nemo research plots for 2024-07-23 nowcast completed
2024-07-23 09:59:22,498 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-23 10:00:16,831 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available
2024-07-23 10:00:17,335 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-23 10:00:17,830 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-23 10:00:18,324 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-23 10:00:18,820 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-23 10:00:19,312 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-23 10:00:19,820 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-23 10:00:20,315 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-23 10:00:20,808 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-23 10:00:21,301 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-23 10:00:21,302 INFO [make_plots] finished rendering nemo nowcast comparison 22jul24 figures
2024-07-23 10:00:21,397 INFO [make_plots] nemo comparison plots for 2024-07-22 nowcast completed
2024-07-23 10:00:21,401 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-23 10:02:53,539 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17807146 = 2024-07-25 01:10:00 UTC, 69.9% complete
2024-07-23 10:07:53,539 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17807619 = 2024-07-25 06:25:20 UTC, 84.5% complete
2024-07-23 10:12:53,539 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17808097 = 2024-07-25 11:44:00 UTC, 99.3% complete
2024-07-23 10:17:53,542 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-07-23 10:17:53,546 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-23 10:17:53,721 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 10:17:53,728 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 10:17:53,732 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:17:53,739 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:17:53,743 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:17:53,749 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:17:53,753 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:17:55,172 INFO [upload_forcing] running in process 743432
2024-07-23 10:17:55,173 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:55,175 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-07-23 10:17:55,176 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:55,177 INFO [upload_forcing] running in process 743430
2024-07-23 10:17:55,178 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:55,180 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-23 10:17:55,181 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:55,182 INFO [upload_forcing] running in process 743429
2024-07-23 10:17:55,183 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:55,185 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-23 10:17:55,186 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:55,193 INFO [upload_forcing] running in process 743431
2024-07-23 10:17:55,194 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:55,195 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-23 10:17:55,196 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:55,201 INFO [download_results] running in process 743433
2024-07-23 10:17:55,202 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:55,203 INFO [download_results] publishing log messages to tcp://*:5570
2024-07-23 10:17:55,205 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:55,206 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/23jul24 to /results/SalishSea/forecast.202111
2024-07-23 10:17:55,799 CRITICAL [upload_forcing] Fraser River turbidity forcing file not found; created symlink to /results/forcing/rivers/river_turb/riverTurbDaily2_y2024m07d22.nc
2024-07-23 10:17:56,052 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 10:17:56,053 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:17:56,056 INFO [upload_forcing] turbidity 2024-07-23 forcing files upload to orcinus-nowcast-agrif completed
2024-07-23 10:17:56,165 INFO [upload_forcing] turbidity 2024-07-23 forcing files upload to arbutus.cloud-nowcast completed
2024-07-23 10:17:56,168 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 10:17:56,188 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:17:56,208 INFO [upload_forcing] turbidity 2024-07-23 forcing files upload to optimum-hindcast completed
2024-07-23 10:17:56,212 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 10:17:56,675 INFO [make_ww3_wind_file] running in process 31685
2024-07-23 10:17:56,676 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:56,677 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-23 10:17:56,714 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:56,715 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-23 forecast run
2024-07-23 10:17:56,742 INFO [make_ww3_current_file] running in process 31684
2024-07-23 10:17:56,743 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:56,744 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-23 10:17:56,745 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:56,746 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-23 forecast run
2024-07-23 10:17:57,178 INFO [make_forcing_links] running in process 743466
2024-07-23 10:17:57,179 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:57,180 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-23 10:17:57,181 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:57,472 INFO [make_forcing_links] running in process 743471
2024-07-23 10:17:57,473 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:57,474 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2024-07-23 10:17:57,475 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:58,000 INFO [make_forcing_links] nowcast-agrif 2024-07-23 forcing file links on orcinus-nowcast-agrif created
2024-07-23 10:17:58,005 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-23 10:17:58,024 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:17:59,264 INFO [run_NEMO_agrif] running in process 743485
2024-07-23 10:17:59,265 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:17:59,266 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5565
2024-07-23 10:17:59,267 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:17:59,286 INFO [upload_forcing] turbidity 2024-07-23 forcing files upload to robot.graham completed
2024-07-23 10:17:59,291 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 10:18:00,705 INFO [make_forcing_links] nowcast-green 2024-07-23 forcing file links on arbutus.cloud-nowcast created
2024-07-23 10:18:00,709 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-23 10:18:00,729 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 10:18:05,026 INFO [run_NEMO] running in process 32445
2024-07-23 10:18:05,027 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:18:05,028 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-23 10:18:05,029 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:18:06,458 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/23jul24nowcast-green_2024-07-23T101805.111643-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-23 10:18:06,479 INFO [run_NEMO] nowcast-green NEMO run for 2024-07-23 on arbutus.cloud-nowcast started
2024-07-23 10:18:06,484 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-23 10:18:06,505 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-07-23 10:18:07,520 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-23 forecast run
2024-07-23 10:18:07,523 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-23 10:18:09,467 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-23 10:18:15,991 INFO [run_NEMO_agrif] job id for 23jul24nowcast-agrif: 11101973.orca2.ibb
2024-07-23 10:18:15,993 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-07-23 queued on orcinus-nowcast-agrif
2024-07-23 10:18:15,998 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-07-23 10:18:16,020 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11101973.orca2.ibb'], host='localhost')
2024-07-23 10:18:17,140 INFO [watch_NEMO_agrif] running in process 743503
2024-07-23 10:18:17,141 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:18:17,142 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5565
2024-07-23 10:18:17,143 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:18:17,901 INFO [watch_NEMO_agrif] watching 23jul24nowcast-agrif job 11101973 on orcinus-nowcast-agrif
2024-07-23 10:18:18,134 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif job 11101973 is queued on orcinus-nowcast-agrif
2024-07-23 10:18:57,186 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-23 10:18:57,189 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-23 forecast run
2024-07-23 10:18:57,211 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 10:18:59,860 INFO [run_ww3] running in process 339
2024-07-23 10:18:59,861 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:18:59,862 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-07-23 10:18:59,863 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:18:59,867 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-07-23T101859.791039-0700
2024-07-23 10:18:59,934 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-07-23T101859.791039-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-07-23 10:18:59,936 INFO [run_ww3] nowcast WaveWatch3 run for 2024-07-23 on arbutus.cloud-nowcast started
2024-07-23 10:19:00,130 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-23 10:19:00,153 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-23 10:19:02,778 INFO [watch_ww3] running in process 425
2024-07-23 10:19:02,778 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:19:02,780 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-07-23 10:19:02,781 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:19:02,933 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-23 10:19:19,179 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-07-23 10:23:09,461 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17802875 = 2024-07-23 01:42:40 UTC, 7.1% complete
2024-07-23 10:24:03,032 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 84, 19.4% complete
2024-07-23 10:24:19,619 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7823678 = 2024-07-23 01:44:40 UTC, 7.3% complete
2024-07-23 10:28:05,088 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 10:28:05,272 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-06'], host='localhost')
2024-07-23 10:28:05,279 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 10:28:05,283 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 10:28:06,498 INFO [download_results] running in process 743540
2024-07-23 10:28:06,499 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:28:06,501 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-23 10:28:06,502 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:28:06,503 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06sep18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 10:28:09,471 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17803056 = 2024-07-23 03:43:20 UTC, 15.5% complete
2024-07-23 10:28:10,713 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 10:28:46,998 INFO [download_results] hindcast 2018-09-06 results files from optimum-hindcast downloaded
2024-07-23 10:28:47,008 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 10:28:47,028 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-06'], host='localhost')
2024-07-23 10:28:48,239 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 10:29:02,936 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 183, 42.4% complete
2024-07-23 10:29:20,059 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7823877 = 2024-07-23 03:57:20 UTC, 16.5% complete
2024-07-23 10:33:09,433 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17803228 = 2024-07-23 05:38:00 UTC, 23.5% complete
2024-07-23 10:34:02,924 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 279, 64.6% complete
2024-07-23 10:34:20,526 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7824077 = 2024-07-23 06:10:40 UTC, 25.8% complete
2024-07-23 10:38:09,501 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17803407 = 2024-07-23 07:37:20 UTC, 31.8% complete
2024-07-23 10:39:03,507 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 372, 86.1% complete
2024-07-23 10:39:20,998 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7824280 = 2024-07-23 08:26:00 UTC, 35.2% complete
2024-07-23 10:43:09,438 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17803591 = 2024-07-23 09:40:00 UTC, 40.3% complete
2024-07-23 10:44:02,979 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-23 10:44:02,982 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-23 10:44:03,153 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:44:03,159 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 10:44:04,334 INFO [download_wwatch3_results] running in process 743620
2024-07-23 10:44:04,335 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:44:04,336 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2024-07-23 10:44:04,337 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:44:06,428 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-23 10:44:06,449 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-23 10:44:11,306 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-23 10:44:21,477 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7824484 = 2024-07-23 10:42:00 UTC, 44.6% complete
2024-07-23 10:48:09,728 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17803767 = 2024-07-23 11:37:20 UTC, 48.4% complete
2024-07-23 10:48:25,513 INFO [download_wwatch3_results] nowcast 2024-07-23 results files from arbutus.cloud-nowcast downloaded
2024-07-23 10:48:25,517 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-23 10:49:11,159 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 84, 13.0% complete
2024-07-23 10:49:21,952 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7824687 = 2024-07-23 12:57:20 UTC, 54.0% complete
2024-07-23 10:49:41,688 INFO [download_results] forecast 2024-07-23 results files from arbutus.cloud-nowcast downloaded
2024-07-23 10:49:41,693 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 10:49:41,714 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:49:43,536 INFO [make_CHS_currents_file] running in process 743679
2024-07-23 10:49:43,537 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:49:43,538 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5567
2024-07-23 10:49:43,539 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:50:24,434 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-23 for forecast
2024-07-23 10:50:24,437 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-23 10:50:24,460 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:50:25,584 INFO [update_forecast_datasets] running in process 743703
2024-07-23 10:50:25,585 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:50:25,586 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2024-07-23 10:50:25,587 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:50:25,588 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/23jul24 run
2024-07-23 10:50:26,024 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/23jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-23 10:50:26,025 INFO [update_forecast_datasets] nemo 2024-07-23 forecast rolling forecast datasets updated
2024-07-23 10:50:26,027 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-23 10:50:26,044 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-23 10:50:26,050 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:50:26,053 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 10:50:27,149 INFO [ping_erddap] running in process 743708
2024-07-23 10:50:27,150 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:50:27,151 INFO [ping_erddap] publishing log messages to tcp://*:5567
2024-07-23 10:50:27,151 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:50:27,160 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-23 10:50:27,162 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 10:50:28,198 INFO [make_surface_current_tiles] running in process 743711
2024-07-23 10:50:28,199 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:50:28,200 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5567
2024-07-23 10:50:28,201 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:50:28,203 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-23 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/23jul24
2024-07-23 10:50:29,907 INFO [make_plots] running in process 743710
2024-07-23 10:50:29,910 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 10:50:29,915 INFO [make_plots] publishing log messages to tcp://*:5568
2024-07-23 10:50:29,916 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-23 10:50:30,572 INFO [make_plots] preparing render list for 2024-07-23 NEMO forecast publish figures
2024-07-23 10:50:30,697 INFO [make_plots] added Website_thumbnail figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,698 INFO [make_plots] added Threshold_website figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,699 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,699 INFO [make_plots] added Boundary Bay figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,700 INFO [make_plots] added Campbell River figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,701 INFO [make_plots] added Cherry Point figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,702 INFO [make_plots] added Friday Harbor figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,703 INFO [make_plots] added Halfmoon Bay figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,703 INFO [make_plots] added Nanaimo figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,704 INFO [make_plots] added Neah Bay figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,704 INFO [make_plots] added New Westminster figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,705 INFO [make_plots] added Patricia Bay figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,705 INFO [make_plots] added Point Atkinson figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,706 INFO [make_plots] added Port Renfrew figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,707 INFO [make_plots] added Sand Heads figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,707 INFO [make_plots] added Sandy Cove figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,708 INFO [make_plots] added Squamish figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,709 INFO [make_plots] added Victoria figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,709 INFO [make_plots] added Woodwards Landing figure to 2024-07-23 NEMO forecast publish render list
2024-07-23 10:50:30,710 INFO [make_plots] starting to render nemo forecast publish 23jul24 figures
2024-07-23 10:53:09,425 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17803950 = 2024-07-23 13:39:20 UTC, 56.9% complete
2024-07-23 10:54:11,029 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 174, 26.9% complete
2024-07-23 10:54:22,395 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7824888 = 2024-07-23 15:11:20 UTC, 63.3% complete
2024-07-23 10:58:09,424 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17804132 = 2024-07-23 15:40:40 UTC, 65.4% complete
2024-07-23 10:59:11,030 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 270, 41.7% complete
2024-07-23 10:59:22,838 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7825092 = 2024-07-23 17:27:20 UTC, 72.8% complete
2024-07-23 11:01:13,187 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/23jul24
2024-07-23 11:03:09,425 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17804314 = 2024-07-23 17:42:00 UTC, 73.8% complete
2024-07-23 11:03:48,054 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-23 11:03:48,054 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-23 11:04:11,032 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 363, 56.0% complete
2024-07-23 11:04:22,993 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:04:23,314 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7825295 = 2024-07-23 19:42:40 UTC, 82.2% complete
2024-07-23 11:05:43,031 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/23jul24
2024-07-23 11:05:43,043 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-23 forecast into /results/nowcast-sys/figures/surface_currents/forecast/23jul24
2024-07-23 11:05:43,044 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-23 forecast completed
2024-07-23 11:05:43,339 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-23 11:06:08,366 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:06:08,717 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:06:43,121 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:07:51,761 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:08:09,427 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17804497 = 2024-07-23 19:44:00 UTC, 82.3% complete
2024-07-23 11:08:26,531 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:09:01,823 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:09:11,034 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 453, 69.9% complete
2024-07-23 11:09:23,752 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7825498 = 2024-07-23 21:58:00 UTC, 91.6% complete
2024-07-23 11:09:37,001 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:10:10,996 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:10:45,899 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:11:20,942 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:11:21,301 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:11:55,711 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:12:29,577 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-18 00:00:00Z to 2024-07-25 11:55:00Z
2024-07-23 11:13:04,050 INFO [make_plots] finished rendering nemo forecast publish 23jul24 figures
2024-07-23 11:13:04,052 INFO [make_plots] nemo publish plots for 2024-07-23 forecast completed
2024-07-23 11:13:04,060 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-23 11:13:04,197 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 11:13:07,331 INFO [make_feeds] running in process 745759
2024-07-23 11:13:07,332 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:13:07,333 INFO [make_feeds] publishing log messages to tcp://*:5567
2024-07-23 11:13:07,335 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:13:09,432 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17804678 = 2024-07-23 21:44:40 UTC, 90.6% complete
2024-07-23 11:13:35,178 INFO [make_feeds] ATOM feeds for 2024-07-23 forecast run completed
2024-07-23 11:13:35,181 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-23 11:14:11,036 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 549, 84.7% complete
2024-07-23 11:14:24,178 INFO [watch_NEMO_agrif] 23jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7825679 = 2024-07-23 23:58:40 UTC, 100.0% complete
2024-07-23 11:18:09,432 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17804852 = 2024-07-23 23:40:40 UTC, 98.7% complete
2024-07-23 11:19:11,038 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 642, 99.1% complete
2024-07-23 11:19:24,342 INFO [watch_NEMO_agrif] NEMO AGRIF run 11101973.orca2.ibb on orcinus-nowcast-agrif completed
2024-07-23 11:19:24,347 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-07-23 11:19:24,671 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 11:19:26,285 INFO [download_results] running in process 745796
2024-07-23 11:19:26,286 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:19:26,288 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-23 11:19:26,289 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:19:26,290 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/23jul24 to /results/SalishSea/nowcast-agrif.201702
2024-07-23 11:19:49,451 INFO [download_results] nowcast-agrif 2024-07-23 results files from orcinus-nowcast-agrif downloaded
2024-07-23 11:19:49,459 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 11:19:49,630 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 11:19:53,996 INFO [make_plots] running in process 745804
2024-07-23 11:19:53,997 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:19:53,999 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-23 11:19:54,000 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:19:54,547 INFO [make_plots] preparing render list for 2024-07-23 NEMO nowcast-agrif research figures
2024-07-23 11:19:54,550 INFO [make_plots] added baynes_sound_surface figure to 2024-07-23 NEMO nowcast-agrif research render list
2024-07-23 11:19:54,551 INFO [make_plots] starting to render nemo nowcast-agrif research 23jul24 figures
2024-07-23 11:20:06,095 INFO [make_plots] finished rendering nemo nowcast-agrif research 23jul24 figures
2024-07-23 11:20:06,098 INFO [make_plots] nemo research plots for 2024-07-23 nowcast-agrif completed
2024-07-23 11:20:06,102 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-23 11:23:09,440 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-07-23 11:23:09,444 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-23 11:23:09,760 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 11:23:11,392 INFO [download_results] running in process 745846
2024-07-23 11:23:11,393 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:23:11,394 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-23 11:23:11,396 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:23:11,397 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/23jul24 to /results2/SalishSea/nowcast-green.202111
2024-07-23 11:24:11,734 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-23 11:24:11,738 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-23 11:24:12,076 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 11:24:13,433 INFO [download_wwatch3_results] running in process 745856
2024-07-23 11:24:13,435 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:24:13,436 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2024-07-23 11:24:13,437 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:30:57,408 INFO [download_wwatch3_results] forecast 2024-07-23 results files from arbutus.cloud-nowcast downloaded
2024-07-23 11:30:57,412 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-23 11:30:57,586 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 11:30:59,462 INFO [update_forecast_datasets] running in process 745889
2024-07-23 11:30:59,463 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:30:59,465 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2024-07-23 11:30:59,466 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:30:59,467 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/23jul24 run
2024-07-23 11:30:59,468 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/23jul24 run
2024-07-23 11:30:59,535 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/23jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-23 11:30:59,535 INFO [update_forecast_datasets] wwatch3 2024-07-23 forecast rolling forecast datasets updated
2024-07-23 11:30:59,538 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-23 11:30:59,712 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-23 11:31:00,991 INFO [ping_erddap] running in process 745895
2024-07-23 11:31:00,993 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:31:00,994 INFO [ping_erddap] publishing log messages to tcp://*:5567
2024-07-23 11:31:00,995 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:31:00,997 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-23 11:31:00,999 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 11:31:01,166 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 11:31:05,303 INFO [make_plots] running in process 745900
2024-07-23 11:31:05,304 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:31:05,306 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-23 11:31:05,307 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:31:05,308 INFO [make_plots] preparing render list for 2024-07-23 WaveWatch3 forecast publish figures
2024-07-23 11:31:05,309 INFO [make_plots] added Halibut Bank figure to 2024-07-23 WaveWatch3 forecast publish render list
2024-07-23 11:31:05,311 INFO [make_plots] added Sentry Shoal figure to 2024-07-23 WaveWatch3 forecast publish render list
2024-07-23 11:31:05,312 INFO [make_plots] starting to render wwatch3 forecast publish 23jul24 figures
2024-07-23 11:31:05,682 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-23 11:31:12,518 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-23 11:31:18,715 INFO [make_plots] finished rendering wwatch3 forecast publish 23jul24 figures
2024-07-23 11:31:18,715 INFO [make_plots] wwatch3 publish plots for 2024-07-23 forecast completed
2024-07-23 11:31:18,719 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-23 11:58:15,543 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 11:58:15,924 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-11'], host='localhost')
2024-07-23 11:58:15,932 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 11:58:15,937 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 11:58:17,182 INFO [download_results] running in process 746024
2024-07-23 11:58:17,183 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 11:58:17,184 INFO [download_results] publishing log messages to tcp://*:5568
2024-07-23 11:58:17,186 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 11:58:17,187 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11sep18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 11:58:21,392 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 11:58:59,216 INFO [download_results] hindcast 2018-09-11 results files from optimum-hindcast downloaded
2024-07-23 11:58:59,224 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 11:58:59,392 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-11'], host='localhost')
2024-07-23 11:59:00,739 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 12:00:13,449 INFO [download_results] nowcast-green 2024-07-23 results files from arbutus.cloud-nowcast downloaded
2024-07-23 12:00:13,455 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 12:00:13,629 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 12:00:13,635 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-07-23 12:00:13,638 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 12:00:13,643 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 12:00:13,649 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 12:00:15,135 INFO [ping_erddap] running in process 746073
2024-07-23 12:00:15,136 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 12:00:15,137 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-23 12:00:15,138 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 12:00:15,143 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-07-23 12:00:15,145 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 12:00:16,264 INFO [make_averaged_dataset] running in process 746074
2024-07-23 12:00:16,265 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 12:00:16,266 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2024-07-23 12:00:16,268 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-23 12:00:16,268 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Jul-2024 biology
2024-07-23 12:00:16,270 INFO [make_averaged_dataset] running in process 746076
2024-07-23 12:00:16,271 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 12:00:16,272 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2024-07-23 12:00:16,273 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-23 12:00:16,274 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Jul-2024 physics
2024-07-23 12:00:16,288 INFO [make_averaged_dataset] running in process 746075
2024-07-23 12:00:16,289 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 12:00:16,290 INFO [make_averaged_dataset] publishing log messages to tcp://*:5568
2024-07-23 12:00:16,291 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-23 12:00:16,291 INFO [make_averaged_dataset] creating day-averaged dataset for 23-Jul-2024 chemistry
2024-07-23 12:00:17,718 INFO [make_plots] running in process 746071
2024-07-23 12:00:17,720 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 12:00:17,721 INFO [make_plots] publishing log messages to tcp://*:5569
2024-07-23 12:00:17,722 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-23 12:00:18,180 INFO [make_plots] preparing render list for 2024-07-23 NEMO nowcast-green research figures
2024-07-23 12:00:19,925 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:20,636 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:21,385 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:22,127 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:22,883 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:23,638 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:24,379 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:25,284 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:26,119 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:26,907 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:27,769 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:28,465 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:29,123 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:29,124 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:29,125 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:29,126 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-07-23 NEMO nowcast-green research render list
2024-07-23 12:00:29,127 INFO [make_plots] starting to render nemo nowcast-green research 23jul24 figures
2024-07-23 12:00:49,352 INFO [make_averaged_dataset] day-averaged dataset for 23-Jul-2024 physics created
2024-07-23 12:00:49,355 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-23 12:01:38,963 INFO [make_averaged_dataset] day-averaged dataset for 23-Jul-2024 chemistry created
2024-07-23 12:01:38,966 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-23 12:01:51,340 INFO [make_averaged_dataset] day-averaged dataset for 23-Jul-2024 biology created
2024-07-23 12:01:51,343 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-23 13:08:30,023 INFO [make_plots] finished rendering nemo nowcast-green research 23jul24 figures
2024-07-23 13:08:30,035 INFO [make_plots] nemo research plots for 2024-07-23 nowcast-green completed
2024-07-23 13:08:30,100 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-23 13:23:25,801 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 13:23:26,160 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-16'], host='localhost')
2024-07-23 13:23:26,167 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 13:23:26,172 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 13:23:27,593 INFO [download_results] running in process 746744
2024-07-23 13:23:27,595 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 13:23:27,596 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-23 13:23:27,597 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 13:23:27,598 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16sep18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 13:23:31,854 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 13:24:08,001 INFO [download_results] hindcast 2018-09-16 results files from optimum-hindcast downloaded
2024-07-23 13:24:08,012 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 13:24:08,215 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-16'], host='localhost')
2024-07-23 13:24:09,576 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 14:48:36,187 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 14:48:36,544 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-21'], host='localhost')
2024-07-23 14:48:36,551 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 14:48:36,556 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 14:48:37,815 INFO [download_results] running in process 747034
2024-07-23 14:48:37,816 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 14:48:37,819 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-23 14:48:37,820 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 14:48:37,821 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21sep18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 14:48:41,985 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 14:49:17,575 INFO [download_results] hindcast 2018-09-21 results files from optimum-hindcast downloaded
2024-07-23 14:49:17,583 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 14:49:17,801 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-21'], host='localhost')
2024-07-23 14:49:19,142 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 15:03:19,628 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240723/18/
2024-07-23 15:03:19,629 INFO [collect_weather] 2024-07-23 2.5km weather forecast 18 collection complete
2024-07-23 15:03:19,632 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-23 15:03:19,984 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-07-23 15:03:19,992 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-07-23 15:03:19,996 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-07-23 15:03:20,000 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-07-24'], host='localhost')
2024-07-23 15:03:21,028 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240723/18/
2024-07-23 15:03:21,029 INFO [crop_gribs] 2024-07-23 18 GRIBs cropping complete
2024-07-23 15:03:21,032 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-23 15:03:21,182 INFO [download_weather] running in process 747140
2024-07-23 15:03:21,183 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 15:03:21,183 INFO [download_weather] publishing log messages to tcp://*:5562
2024-07-23 15:03:21,184 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-23 15:03:21,185 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240723
2024-07-23 15:03:21,186 INFO [download_weather] running in process 747142
2024-07-23 15:03:21,187 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 15:03:21,188 INFO [download_weather] publishing log messages to tcp://*:5564
2024-07-23 15:03:21,188 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-23 15:03:21,190 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240723
2024-07-23 15:03:21,245 INFO [collect_weather] running in process 747143
2024-07-23 15:03:21,246 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 15:03:21,247 INFO [collect_weather] publishing log messages to tcp://*:5567
2024-07-23 15:03:21,248 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-23 15:03:21,309 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-07-23 15:03:22,406 INFO [crop_gribs] running in process 747144
2024-07-23 15:03:22,407 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 15:03:22,408 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-07-23 15:03:22,409 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-23 15:03:22,410 INFO [crop_gribs] cropping 2024-07-24 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-07-23 15:03:22,423 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240724/00/
2024-07-23 15:08:08,265 INFO [download_weather] 2024-07-23 1km weather forecast 12 downloads complete
2024-07-23 15:08:08,268 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-23 15:08:19,048 INFO [download_weather] 2024-07-23 1km weather forecast 00 downloads complete
2024-07-23 15:08:19,051 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-23 16:18:46,923 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 16:18:47,305 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-09-26'], host='localhost')
2024-07-23 16:18:47,314 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 16:18:47,319 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 16:18:48,711 INFO [download_results] running in process 747541
2024-07-23 16:18:48,712 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 16:18:48,714 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-23 16:18:48,715 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 16:18:48,717 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26sep18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 16:18:52,940 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 16:19:33,343 INFO [download_results] hindcast 2018-09-26 results files from optimum-hindcast downloaded
2024-07-23 16:19:33,352 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 16:19:33,566 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-09-26'], host='localhost')
2024-07-23 16:19:35,289 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 16:19:35,517 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-sep', 'robot.graham'], host='localhost')
2024-07-23 16:19:36,733 INFO [archive_tarball] running in process 747590
2024-07-23 16:19:36,734 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 16:19:36,735 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-07-23 16:19:36,736 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-23 16:19:36,737 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-sep18.tar from /ocean/sallen/timbucktwo/oxygen/*sep18/
2024-07-23 16:25:29,071 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-sep18.index from /ocean/dlatorne/oxygen-sep18.tar
2024-07-23 16:25:29,151 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-sep18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-23 17:29:51,061 INFO [archive_tarball] hindcast *sep18 results files archived to robot.graham
2024-07-23 17:29:51,064 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-23 17:43:57,811 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 17:43:58,226 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-01'], host='localhost')
2024-07-23 17:43:58,234 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 17:43:58,240 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 17:43:59,756 INFO [download_results] running in process 748372
2024-07-23 17:43:59,758 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 17:43:59,759 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-23 17:43:59,761 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 17:43:59,772 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 17:44:03,901 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 17:44:45,705 INFO [download_results] hindcast 2018-10-01 results files from optimum-hindcast downloaded
2024-07-23 17:44:45,716 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 17:44:45,939 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-01'], host='localhost')
2024-07-23 17:44:47,459 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 19:09:08,328 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 19:09:08,688 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-06'], host='localhost')
2024-07-23 19:09:08,696 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 19:09:08,701 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 19:09:09,946 INFO [download_results] running in process 748936
2024-07-23 19:09:09,947 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 19:09:09,948 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-23 19:09:09,950 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 19:09:09,951 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 19:09:14,146 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 19:09:49,773 INFO [download_results] hindcast 2018-10-06 results files from optimum-hindcast downloaded
2024-07-23 19:09:49,783 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 19:09:49,983 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-06'], host='localhost')
2024-07-23 19:09:51,287 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 20:39:19,094 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 20:39:19,452 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-11'], host='localhost')
2024-07-23 20:39:19,459 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 20:39:19,463 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 20:39:20,713 INFO [download_results] running in process 749572
2024-07-23 20:39:20,715 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 20:39:20,716 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-23 20:39:20,717 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 20:39:20,718 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 20:39:24,780 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 20:40:00,547 INFO [download_results] hindcast 2018-10-11 results files from optimum-hindcast downloaded
2024-07-23 20:40:00,558 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 20:40:00,763 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-11'], host='localhost')
2024-07-23 20:40:02,011 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 21:03:43,061 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240724/00/
2024-07-23 21:03:43,062 INFO [collect_weather] 2024-07-24 2.5km weather forecast 00 collection complete
2024-07-23 21:03:43,065 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-23 21:03:43,423 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-07-23 21:03:43,430 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-07-24'], host='localhost')
2024-07-23 21:03:43,456 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240724/00/
2024-07-23 21:03:43,457 INFO [crop_gribs] 2024-07-24 00 GRIBs cropping complete
2024-07-23 21:03:43,460 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-23 21:03:44,658 INFO [collect_weather] running in process 749671
2024-07-23 21:03:44,659 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 21:03:44,660 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-23 21:03:44,661 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-23 21:03:44,679 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-07-23 21:03:45,391 INFO [crop_gribs] running in process 749673
2024-07-23 21:03:45,392 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 21:03:45,394 INFO [crop_gribs] publishing log messages to tcp://*:5564
2024-07-23 21:03:45,395 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-23 21:03:45,396 INFO [crop_gribs] cropping 2024-07-24 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-07-23 21:03:45,410 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240724/06/
2024-07-23 22:04:29,314 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 22:04:29,685 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-16'], host='localhost')
2024-07-23 22:04:29,692 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 22:04:29,698 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 22:04:30,924 INFO [download_results] running in process 749994
2024-07-23 22:04:30,925 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 22:04:30,926 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-23 22:04:30,927 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 22:04:30,929 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 22:04:35,269 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 22:05:10,656 INFO [download_results] hindcast 2018-10-16 results files from optimum-hindcast downloaded
2024-07-23 22:05:10,666 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 22:05:10,870 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-16'], host='localhost')
2024-07-23 22:05:12,054 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 23:29:39,657 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 23:29:40,077 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-21'], host='localhost')
2024-07-23 23:29:40,084 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 23:29:40,090 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 23:29:41,307 INFO [download_results] running in process 750322
2024-07-23 23:29:41,308 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 23:29:41,310 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-23 23:29:41,311 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 23:29:41,312 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 23:29:45,566 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 23:30:21,354 INFO [download_results] hindcast 2018-10-21 results files from optimum-hindcast downloaded
2024-07-23 23:30:21,364 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 23:30:21,531 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-21'], host='localhost')
2024-07-23 23:30:22,865 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 01:14:51,893 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 01:14:52,240 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-26'], host='localhost')
2024-07-24 01:14:52,248 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 01:14:52,253 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 01:14:53,458 INFO [download_results] running in process 750739
2024-07-24 01:14:53,459 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 01:14:53,461 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-24 01:14:53,462 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 01:14:53,463 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 01:14:57,727 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 01:15:38,568 INFO [download_results] hindcast 2018-10-26 results files from optimum-hindcast downloaded
2024-07-24 01:15:38,578 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 01:15:38,778 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-26'], host='localhost')
2024-07-24 01:15:39,991 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 01:15:40,199 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-oct', 'robot.graham'], host='localhost')
2024-07-24 01:15:41,474 INFO [archive_tarball] running in process 750788
2024-07-24 01:15:41,476 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 01:15:41,477 INFO [archive_tarball] publishing log messages to tcp://*:5565
2024-07-24 01:15:41,478 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-24 01:15:41,480 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct18.tar from /ocean/sallen/timbucktwo/oxygen/*oct18/
2024-07-24 01:21:27,205 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct18.index from /ocean/dlatorne/oxygen-oct18.tar
2024-07-24 01:21:27,294 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-oct18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-24 01:28:51,027 INFO [archive_tarball] hindcast *oct18 results files archived to robot.graham
2024-07-24 01:28:51,031 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-24 02:40:02,086 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 02:40:02,451 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-01'], host='localhost')
2024-07-24 02:40:02,459 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 02:40:02,465 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 02:40:03,666 INFO [download_results] running in process 751087
2024-07-24 02:40:03,667 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 02:40:03,669 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-24 02:40:03,670 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 02:40:03,671 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 02:40:07,943 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 02:40:43,813 INFO [download_results] hindcast 2018-11-01 results files from optimum-hindcast downloaded
2024-07-24 02:40:43,823 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 02:40:44,028 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-01'], host='localhost')
2024-07-24 02:40:45,244 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 03:03:38,432 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240724/06/
2024-07-24 03:03:38,434 INFO [collect_weather] 2024-07-24 2.5km weather forecast 06 collection complete
2024-07-24 03:03:38,437 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-24 03:03:38,791 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,798 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,804 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,808 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,812 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,816 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,821 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,827 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,835 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,845 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,855 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,863 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,879 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,884 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,890 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-07-24 03:03:38,905 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-07-24 03:03:38,918 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-07-24 03:03:38,931 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-07-23'], host='localhost')
2024-07-24 03:03:38,947 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-07-24 03:03:38,959 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-07-24 03:03:38,980 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-07-24 03:03:38,996 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240724/06/
2024-07-24 03:03:38,997 INFO [crop_gribs] 2024-07-24 06 GRIBs cropping complete
2024-07-24 03:03:39,000 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-24 03:03:39,710 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-07-24 03:03:41,926 INFO [collect_weather] running in process 751206
2024-07-24 03:03:41,927 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:41,928 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-24 03:03:41,930 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:41,960 INFO [collect_NeahBay_ssh] running in process 751205
2024-07-24 03:03:41,961 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:41,961 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-07-24 03:03:41,962 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:41,963 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-07-24
2024-07-24 03:03:41,995 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-07-24 03:03:42,706 INFO [collect_river_data] running in process 751191
2024-07-24 03:03:42,707 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,707 INFO [collect_river_data] publishing log messages to tcp://*:5565
2024-07-24 03:03:42,709 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,710 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-07-23
2024-07-24 03:03:42,867 INFO [collect_river_data] running in process 751188
2024-07-24 03:03:42,868 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,870 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-24 03:03:42,871 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,871 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-07-23
2024-07-24 03:03:42,878 INFO [collect_river_data] running in process 751189
2024-07-24 03:03:42,879 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,881 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-24 03:03:42,884 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,886 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-07-23
2024-07-24 03:03:42,887 INFO [collect_river_data] running in process 751200
2024-07-24 03:03:42,887 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,888 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-24 03:03:42,889 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,889 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-07-23
2024-07-24 03:03:42,890 INFO [collect_river_data] running in process 751195
2024-07-24 03:03:42,891 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,892 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-24 03:03:42,893 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,894 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-07-23
2024-07-24 03:03:42,927 INFO [collect_river_data] running in process 751192
2024-07-24 03:03:42,928 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,929 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-24 03:03:42,931 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,932 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-07-23
2024-07-24 03:03:42,948 INFO [collect_river_data] running in process 751194
2024-07-24 03:03:42,949 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,950 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-07-24 03:03:42,951 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,952 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-07-23
2024-07-24 03:03:42,962 INFO [collect_river_data] running in process 751185
2024-07-24 03:03:42,963 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,964 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-07-24 03:03:42,967 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,972 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-07-23
2024-07-24 03:03:42,976 INFO [collect_river_data] running in process 751190
2024-07-24 03:03:42,982 INFO [collect_river_data] running in process 751199
2024-07-24 03:03:42,984 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,985 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,986 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-07-24 03:03:42,986 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-07-24 03:03:42,987 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,988 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:42,990 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-07-23
2024-07-24 03:03:42,991 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-07-23
2024-07-24 03:03:42,994 INFO [collect_river_data] running in process 751187
2024-07-24 03:03:42,994 INFO [collect_river_data] running in process 751196
2024-07-24 03:03:42,995 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,995 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:42,997 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-07-24 03:03:42,998 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-07-24 03:03:42,999 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:43,000 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:43,002 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-07-23
2024-07-24 03:03:43,003 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-07-23
2024-07-24 03:03:43,013 INFO [collect_river_data] running in process 751198
2024-07-24 03:03:43,014 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:43,015 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-07-24 03:03:43,016 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:43,017 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-07-23
2024-07-24 03:03:43,041 INFO [collect_river_data] running in process 751197
2024-07-24 03:03:43,043 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:43,044 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-07-24 03:03:43,044 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:43,045 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-07-23
2024-07-24 03:03:43,316 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-07-24 03:03:43,317 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-07-23 completed
2024-07-24 03:03:43,319 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:43,324 INFO [get_vfpa_hadcp] running in process 751204
2024-07-24 03:03:43,325 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:43,326 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5590
2024-07-24 03:03:43,328 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:43,329 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-23
2024-07-24 03:03:43,518 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-07-23 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-07-24 03:03:43,518 INFO [collect_river_data] ECCC Englishman river data collection for 2024-07-23 completed
2024-07-24 03:03:43,540 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-07-24 03:03:43,541 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-07-23 completed
2024-07-24 03:03:43,645 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-07-24 03:03:43,645 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-07-23 completed
2024-07-24 03:03:43,658 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-07-24 03:03:43,660 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-07-23 completed
2024-07-24 03:03:43,708 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-07-24 03:03:43,709 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-07-23 completed
2024-07-24 03:03:43,734 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-07-24 03:03:43,735 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-07-23 completed
2024-07-24 03:03:43,736 INFO [crop_gribs] running in process 751207
2024-07-24 03:03:43,736 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:43,737 INFO [crop_gribs] publishing log messages to tcp://*:5591
2024-07-24 03:03:43,739 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:43,740 INFO [crop_gribs] cropping 2024-07-24 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-07-24 03:03:43,754 INFO [collect_NeahBay_ssh] 2024-07-24T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-07-24 03:03:43,763 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240724/12/
2024-07-24 03:03:43,793 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-07-24 03:03:43,794 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-07-23 completed
2024-07-24 03:03:43,797 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-07-24 03:03:43,797 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-07-24 03:03:43,798 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-07-23 completed
2024-07-24 03:03:43,799 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-07-23 completed
2024-07-24 03:03:43,850 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-07-23 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-07-24 03:03:43,851 INFO [collect_river_data] ECCC Fraser river data collection for 2024-07-23 completed
2024-07-24 03:03:43,853 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-07-24 03:03:43,853 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-07-23 completed
2024-07-24 03:03:43,857 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-07-23 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-07-24 03:03:43,858 INFO [collect_river_data] ECCC Capilano river data collection for 2024-07-23 completed
2024-07-24 03:03:43,861 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-07-23 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-07-24 03:03:43,864 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-07-23 completed
2024-07-24 03:03:43,867 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:44,027 INFO [get_onc_ferry] running in process 751203
2024-07-24 03:03:44,028 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:44,029 INFO [get_onc_ferry] publishing log messages to tcp://*:5592
2024-07-24 03:03:44,029 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:44,030 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-07-23
2024-07-24 03:03:44,069 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:44,124 INFO [grib_to_netcdf] running in process 751263
2024-07-24 03:03:44,125 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:44,125 INFO [grib_to_netcdf] publishing log messages to tcp://*:5593
2024-07-24 03:03:44,126 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:44,127 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-24 forecast2 run
2024-07-24 03:03:44,131 INFO [get_onc_ctd] running in process 751201
2024-07-24 03:03:44,133 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:44,133 INFO [get_onc_ctd] publishing log messages to tcp://*:5594
2024-07-24 03:03:44,134 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:44,135 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-07-23
2024-07-24 03:03:44,274 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:44,477 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:44,680 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:44,879 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:45,065 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-24 03:03:45,260 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 03:03:45,266 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:45,512 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:45,731 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-23 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202407.nc
2024-07-24 03:03:45,731 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-07 netcdf file
2024-07-24 03:03:45,739 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:45,793 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-07-23
2024-07-24 03:03:45,928 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:46,120 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:46,210 CRITICAL [get_onc_ctd] 2024-07-23 ONC SCVIP CTD T&S file creation failed
2024-07-24 03:03:46,622 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:46,624 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-24 03:03:46,717 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2024-07-24 03:03:46,900 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2024-07-24 03:03:47,978 INFO [ping_erddap] running in process 751684
2024-07-24 03:03:47,979 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:47,980 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-24 03:03:47,982 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:47,984 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2024-07-24 03:03:47,986 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-24 03:03:48,016 INFO [make_ssh_files] running in process 751650
2024-07-24 03:03:48,017 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:48,019 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2024-07-24 03:03:48,020 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:48,021 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-07-24 NOAA Neah Bay 00Z observation and forecast values
2024-07-24 03:03:52,461 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc
2024-07-24 03:03:52,635 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-24 03:03:52,651 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d25.nc
2024-07-24 03:03:52,808 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-24 03:03:52,993 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d22.nc
2024-07-24 03:03:53,175 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d23.nc
2024-07-24 03:03:53,362 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d24.nc
2024-07-24 03:03:53,542 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d25.nc
2024-07-24 03:03:53,726 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d26.nc
2024-07-24 03:03:53,906 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d27.nc
2024-07-24 03:03:54,035 INFO [make_ssh_files] sea surface height boundary file for 2024-07-24 forecast2 run created
2024-07-24 03:03:54,039 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-24 03:03:54,253 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-24 03:03:54,257 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-24 03:03:54,956 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d26.nc
2024-07-24 03:03:54,962 INFO [grib_to_netcdf] 2024-07-24 NEMO-atmos forcing file for forecast2 created
2024-07-24 03:03:54,964 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-24 03:03:55,570 INFO [make_runoff_file] running in process 751717
2024-07-24 03:03:55,571 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:55,572 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-07-24 03:03:55,573 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:55,604 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-24 03:03:55,674 INFO [make_runoff_file] 2024-07-24 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-24 03:03:55,676 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-24 03:03:56,140 INFO [make_v202111_runoff_file] running in process 751715
2024-07-24 03:03:56,141 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:03:56,142 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5565
2024-07-24 03:03:56,143 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:03:56,144 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-23
2024-07-24 03:03:56,957 ERROR [make_v202111_runoff_file] no 2024-07-23 discharge obs for Skagit_MountVernon: patching
2024-07-24 03:03:57,485 ERROR [make_v202111_runoff_file] no 2024-07-23 discharge obs for Snohomish_Monroe: patching
2024-07-24 03:03:57,668 ERROR [make_v202111_runoff_file] no 2024-07-23 discharge obs for Nisqually_McKenna: patching
2024-07-24 03:03:57,749 ERROR [make_v202111_runoff_file] no 2024-07-23 discharge obs for Greenwater_Greenwater: patching
2024-07-24 03:03:58,957 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-23: /results/forcing/rivers/R202108Dailies_y2024m07d23.nc
2024-07-24 03:03:58,958 INFO [make_v202111_runoff_file] 2024-07-23 runoff file creation completed
2024-07-24 03:03:58,960 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-24 03:03:59,166 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-07-24 03:03:59,170 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-07-24 03:03:59,174 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-07-24 03:03:59,177 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-07-24 03:04:00,341 INFO [upload_forcing] running in process 751758
2024-07-24 03:04:00,342 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:04:00,343 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-07-24 03:04:00,344 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:04:00,348 INFO [upload_forcing] running in process 751759
2024-07-24 03:04:00,349 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:04:00,350 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-07-24 03:04:00,351 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:04:00,365 INFO [upload_forcing] running in process 751757
2024-07-24 03:04:00,366 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:04:00,367 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-24 03:04:00,368 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:04:00,373 INFO [upload_forcing] running in process 751756
2024-07-24 03:04:00,374 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:04:00,375 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-24 03:04:00,376 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:04:01,718 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d23.nc
2024-07-24 03:04:01,750 INFO [upload_forcing] forecast2 2024-07-24 forcing files upload to optimum-hindcast completed
2024-07-24 03:04:01,757 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 03:04:02,005 INFO [upload_forcing] forecast2 2024-07-24 forcing files upload to orcinus-nowcast-agrif completed
2024-07-24 03:04:02,008 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 03:04:02,629 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-07-23
2024-07-24 03:04:03,243 INFO [upload_forcing] forecast2 2024-07-24 forcing files upload to arbutus.cloud-nowcast completed
2024-07-24 03:04:03,247 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 03:04:03,451 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 03:04:03,645 WARNING [get_onc_ferry] No ONC TWDP TSG temperature,conductivity,salinity data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:03,652 WARNING [get_onc_ferry] No ONC TWDP TSG temperature data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:03,653 WARNING [get_onc_ferry] No ONC TWDP TSG conductivity data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:03,655 WARNING [get_onc_ferry] No ONC TWDP TSG salinity data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:03,656 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-07-23
2024-07-24 03:04:04,435 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation,oxygen_corrected,temperature data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:04,441 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:04,443 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:04,445 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR temperature data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:04,446 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-07-23
2024-07-24 03:04:04,661 INFO [make_forcing_links] running in process 751786
2024-07-24 03:04:04,662 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:04:04,663 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-24 03:04:04,664 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:04:05,070 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence,chlorophyll,turbidity data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:05,076 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:05,078 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL chlorophyll data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:05,080 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL turbidity data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:05,081 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-07-23
2024-07-24 03:04:05,835 INFO [make_forcing_links] forecast2 2024-07-24 forcing file links on arbutus.cloud-nowcast created
2024-07-24 03:04:05,839 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-24 03:04:06,053 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 03:04:06,305 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:06,309 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:06,311 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:06,312 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-07-23
2024-07-24 03:04:08,155 INFO [run_NEMO] running in process 11232
2024-07-24 03:04:08,155 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:04:08,156 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-24 03:04:08,261 INFO [upload_forcing] forecast2 2024-07-24 forcing files upload to robot.graham completed
2024-07-24 03:04:08,265 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-24 03:04:16,153 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:04:17,151 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/23jul24forecast2_2024-07-24T030416.227747-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-24 03:04:17,170 INFO [run_NEMO] forecast2 NEMO run for 2024-07-24 on arbutus.cloud-nowcast started
2024-07-24 03:04:17,174 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-24 03:04:17,383 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-24 03:04:19,389 INFO [watch_NEMO] running in process 11486
2024-07-24 03:04:19,391 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:04:19,392 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-24 03:04:19,393 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:04:19,463 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-24 03:04:23,071 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-07-23
2024-07-24 03:04:31,620 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-07-23
2024-07-24 03:04:39,660 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:39,661 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-07-23
2024-07-24 03:04:47,505 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-23; substituting empty dataset
2024-07-24 03:04:47,507 WARNING [get_onc_ferry] ONC TWDP TSG temperature data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,529 WARNING [get_onc_ferry] ONC TWDP TSG conductivity data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,592 WARNING [get_onc_ferry] ONC TWDP TSG salinity data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,610 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,628 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,646 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR temperature data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,663 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,681 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL chlorophyll data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,698 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL turbidity data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,715 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,733 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,834 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:04:47,851 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-23 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-24 03:05:53,851 INFO [get_onc_ferry] 2024-07-23 ONC TWDP ferry data file created
2024-07-24 03:05:53,854 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-07-24 03:05:54,058 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-07-24 03:05:55,382 INFO [ping_erddap] running in process 751804
2024-07-24 03:05:55,383 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:05:55,384 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-24 03:05:55,386 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:05:55,387 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-07-24 03:05:55,390 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-24 03:09:19,474 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17807494 = 2024-07-25 05:02:00 UTC, 16.8% complete
2024-07-24 03:14:19,465 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17808020 = 2024-07-25 10:52:40 UTC, 36.3% complete
2024-07-24 03:19:19,466 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17808555 = 2024-07-25 16:49:20 UTC, 56.1% complete
2024-07-24 03:24:19,467 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17809037 = 2024-07-25 22:10:40 UTC, 74.0% complete
2024-07-24 03:29:19,468 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17809502 = 2024-07-26 03:20:40 UTC, 91.2% complete
2024-07-24 03:34:19,473 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-07-24 03:34:19,477 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-24 03:34:19,832 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 03:34:19,841 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 03:34:19,845 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-24 03:34:21,489 INFO [download_results] running in process 751880
2024-07-24 03:34:21,490 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:34:21,492 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-24 03:34:21,493 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:34:21,494 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/23jul24 to /results/SalishSea/forecast2.202111
2024-07-24 03:34:22,628 INFO [make_ww3_wind_file] running in process 11912
2024-07-24 03:34:22,629 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:34:22,630 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-24 03:34:22,632 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:34:22,633 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-24 forecast2 run
2024-07-24 03:34:22,751 INFO [make_ww3_current_file] running in process 11914
2024-07-24 03:34:22,752 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:34:22,754 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-24 03:34:22,755 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:34:22,756 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-24 forecast2 run
2024-07-24 03:34:31,720 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-24 forecast2 run
2024-07-24 03:34:31,723 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-24 03:36:30,132 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-24 forecast2 run
2024-07-24 03:36:30,137 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-24 03:36:30,352 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-24'], host='arbutus.cloud-nowcast')
2024-07-24 03:36:35,903 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-24 03:36:36,117 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-24 03:36:41,488 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-24 03:41:41,563 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 75, 13.9% complete
2024-07-24 03:46:41,745 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 165, 30.6% complete
2024-07-24 03:51:41,915 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 246, 45.6% complete
2024-07-24 03:56:41,844 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 345, 63.9% complete
2024-07-24 03:59:13,787 INFO [download_results] forecast2 2024-07-23 results files from arbutus.cloud-nowcast downloaded
2024-07-24 03:59:13,790 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 03:59:13,994 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-24 03:59:16,509 INFO [make_CHS_currents_file] running in process 751933
2024-07-24 03:59:16,511 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:59:16,512 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-24 03:59:16,513 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:59:51,036 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-23 for forecast2
2024-07-24 03:59:51,039 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-24 03:59:51,244 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-24 03:59:52,464 INFO [update_forecast_datasets] running in process 751957
2024-07-24 03:59:52,465 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 03:59:52,466 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2024-07-24 03:59:52,468 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-24 03:59:52,469 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/23jul24 run
2024-07-24 04:01:41,258 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 426, 78.9% complete
2024-07-24 04:04:01,342 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/23jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-24 04:04:01,344 INFO [update_forecast_datasets] nemo 2024-07-23 forecast2 rolling forecast datasets updated
2024-07-24 04:04:01,346 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-24 04:04:01,516 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-24 04:04:01,522 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-07-23'], host='localhost')
2024-07-24 04:04:01,526 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-24 04:04:02,802 INFO [ping_erddap] running in process 752033
2024-07-24 04:04:02,803 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:04:02,804 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-24 04:04:02,805 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:04:02,815 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-24 04:04:02,817 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-24 04:04:04,077 INFO [make_surface_current_tiles] running in process 752036
2024-07-24 04:04:04,078 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:04:04,079 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2024-07-24 04:04:04,081 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:04:04,120 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-23 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/23jul24
2024-07-24 04:04:06,173 INFO [make_plots] running in process 752035
2024-07-24 04:04:06,174 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:04:06,175 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-24 04:04:06,176 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:04:06,831 INFO [make_plots] preparing render list for 2024-07-23 NEMO forecast2 publish figures
2024-07-24 04:04:06,953 INFO [make_plots] added Website_thumbnail figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,954 INFO [make_plots] added Threshold_website figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,955 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,956 INFO [make_plots] added Boundary Bay figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,956 INFO [make_plots] added Campbell River figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,957 INFO [make_plots] added Cherry Point figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,958 INFO [make_plots] added Friday Harbor figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,959 INFO [make_plots] added Halfmoon Bay figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,960 INFO [make_plots] added Nanaimo figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,961 INFO [make_plots] added Neah Bay figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,962 INFO [make_plots] added New Westminster figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,962 INFO [make_plots] added Patricia Bay figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,963 INFO [make_plots] added Point Atkinson figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,964 INFO [make_plots] added Port Renfrew figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,965 INFO [make_plots] added Sand Heads figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,966 INFO [make_plots] added Sandy Cove figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,967 INFO [make_plots] added Squamish figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,968 INFO [make_plots] added Victoria figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,968 INFO [make_plots] added Woodwards Landing figure to 2024-07-23 NEMO forecast2 publish render list
2024-07-24 04:04:06,969 INFO [make_plots] starting to render nemo forecast2 publish 23jul24 figures
2024-07-24 04:05:12,237 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-24 04:05:12,785 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-06'], host='localhost')
2024-07-24 04:05:12,791 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 04:05:12,802 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-24 04:05:14,532 INFO [download_results] running in process 752370
2024-07-24 04:05:14,533 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:05:14,534 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-24 04:05:14,535 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:05:14,536 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-24 04:05:18,874 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-24 04:06:00,041 INFO [download_results] hindcast 2018-11-06 results files from optimum-hindcast downloaded
2024-07-24 04:06:00,051 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-24 04:06:00,456 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-06'], host='localhost')
2024-07-24 04:06:02,032 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-24 04:06:41,261 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 522, 96.7% complete
2024-07-24 04:11:41,263 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-24 04:11:41,265 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-24 04:11:41,728 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 04:11:43,049 INFO [download_wwatch3_results] running in process 753419
2024-07-24 04:11:43,050 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:11:43,051 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-07-24 04:11:43,052 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:14:02,846 INFO [download_wwatch3_results] forecast2 2024-07-24 results files from arbutus.cloud-nowcast downloaded
2024-07-24 04:14:02,849 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-24 04:14:03,256 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 04:14:04,954 INFO [update_forecast_datasets] running in process 753880
2024-07-24 04:14:04,955 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:14:04,956 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566
2024-07-24 04:14:04,957 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:14:04,958 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/24jul24 run
2024-07-24 04:14:04,959 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/24jul24 run
2024-07-24 04:14:38,547 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/24jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-24 04:14:38,547 INFO [update_forecast_datasets] wwatch3 2024-07-24 forecast2 rolling forecast datasets updated
2024-07-24 04:14:38,550 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-24 04:14:38,957 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-24 04:14:40,393 INFO [ping_erddap] running in process 753979
2024-07-24 04:14:40,394 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:14:40,395 INFO [ping_erddap] publishing log messages to tcp://*:5566
2024-07-24 04:14:40,397 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:14:40,398 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-24 04:14:40,399 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-24 04:14:40,677 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-07-24'], host='localhost')
2024-07-24 04:14:45,977 INFO [make_plots] running in process 753998
2024-07-24 04:14:45,980 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:14:45,981 INFO [make_plots] publishing log messages to tcp://*:5566
2024-07-24 04:14:45,982 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:14:45,982 INFO [make_plots] preparing render list for 2024-07-24 WaveWatch3 forecast2 publish figures
2024-07-24 04:14:45,983 INFO [make_plots] added Halibut Bank figure to 2024-07-24 WaveWatch3 forecast2 publish render list
2024-07-24 04:14:45,984 INFO [make_plots] added Sentry Shoal figure to 2024-07-24 WaveWatch3 forecast2 publish render list
2024-07-24 04:14:45,985 INFO [make_plots] starting to render wwatch3 forecast2 publish 24jul24 figures
2024-07-24 04:14:46,472 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-24 04:14:54,318 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-24 04:15:00,875 INFO [make_plots] finished rendering wwatch3 forecast2 publish 24jul24 figures
2024-07-24 04:15:00,876 INFO [make_plots] wwatch3 publish plots for 2024-07-24 forecast2 completed
2024-07-24 04:15:00,879 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-24 04:15:29,963 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/23jul24
2024-07-24 04:17:59,583 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-24 04:17:59,584 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-24 04:18:35,647 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:19:39,929 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/23jul24
2024-07-24 04:19:39,940 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-23 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/23jul24
2024-07-24 04:19:39,941 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-23 forecast2 completed
2024-07-24 04:19:40,237 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-24 04:20:24,541 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:20:24,895 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:21:00,656 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:22:11,373 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:22:47,162 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:23:22,965 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:23:58,643 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:24:33,887 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:25:09,713 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:25:45,866 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:25:46,223 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:26:22,123 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:26:57,262 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-19 00:00:00Z to 2024-07-26 05:55:00Z
2024-07-24 04:27:32,818 INFO [make_plots] finished rendering nemo forecast2 publish 23jul24 figures
2024-07-24 04:27:32,821 INFO [make_plots] nemo publish plots for 2024-07-23 forecast2 completed
2024-07-24 04:27:32,830 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-24 04:27:33,160 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-24 04:27:36,457 INFO [make_feeds] running in process 754138
2024-07-24 04:27:36,459 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:27:36,460 INFO [make_feeds] publishing log messages to tcp://*:5564
2024-07-24 04:27:36,461 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:28:04,660 INFO [make_feeds] ATOM feeds for 2024-07-23 forecast2 run completed
2024-07-24 04:28:04,663 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-24 04:28:04,938 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-07-24 04:28:06,121 INFO [clear_checklist] running in process 754162
2024-07-24 04:28:06,123 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:28:06,124 INFO [clear_checklist] publishing log messages to tcp://*:5564
2024-07-24 04:28:06,125 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:28:06,126 INFO [clear_checklist] requesting that manager clear system state checklist
2024-07-24 04:28:06,129 INFO [manager] writing checklist to log file
2024-07-24 04:28:06,172 INFO [manager] checklist cleared
2024-07-24 04:28:06,174 INFO [clear_checklist] nowcast system checklist cleared
2024-07-24 04:28:06,177 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-07-24 04:28:07,275 INFO [rotate_logs] running in process 754167
2024-07-24 04:28:07,276 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-24 04:28:07,278 INFO [rotate_logs] publishing log messages to tcp://*:5564
2024-07-24 04:28:07,279 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-07-24 04:28:07,280 INFO [rotate_logs] rotating log files