2024-04-24 04:45:08,273 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-04-24 04:45:08,280 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-04-24 04:45:08,285 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-04-24 04:45:08,286 INFO [rotate_logs] log files rotated
2024-04-24 04:45:08,334 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-04-24 05:44:26,645 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 05:44:26,916 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-08-16'], host='localhost')
2024-04-24 05:44:26,927 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 05:44:26,940 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 05:44:28,647 INFO [download_results] running in process 2128636
2024-04-24 05:44:28,648 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 05:44:28,649 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-24 05:44:28,650 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 05:44:28,650 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16aug20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 05:44:32,896 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 05:45:18,721 INFO [download_results] hindcast 2020-08-16 results files from optimum-hindcast downloaded
2024-04-24 05:45:18,734 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 05:45:18,743 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-08-16'], host='localhost')
2024-04-24 05:45:19,634 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 07:14:37,659 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 07:14:37,886 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-08-21'], host='localhost')
2024-04-24 07:14:37,895 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 07:14:37,900 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 07:14:39,979 INFO [download_results] running in process 2129934
2024-04-24 07:14:39,980 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 07:14:39,981 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-24 07:14:39,982 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 07:14:39,982 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21aug20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 07:14:44,073 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 07:15:41,152 INFO [download_results] hindcast 2020-08-21 results files from optimum-hindcast downloaded
2024-04-24 07:15:41,167 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 07:15:41,174 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-08-21'], host='localhost')
2024-04-24 07:15:42,318 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 08:54:50,551 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 08:54:50,819 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-08-26'], host='localhost')
2024-04-24 08:54:50,828 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 08:54:50,832 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 08:54:52,871 INFO [download_results] running in process 2130345
2024-04-24 08:54:52,873 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 08:54:52,873 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-24 08:54:52,874 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 08:54:52,875 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26aug20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 08:54:56,868 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 08:56:03,289 INFO [download_results] hindcast 2020-08-26 results files from optimum-hindcast downloaded
2024-04-24 08:56:03,303 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 08:56:03,314 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-08-26'], host='localhost')
2024-04-24 08:56:04,401 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 08:56:04,410 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2020-aug', 'robot.graham'], host='localhost')
2024-04-24 08:56:05,352 INFO [archive_tarball] running in process 2130396
2024-04-24 08:56:05,353 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 08:56:05,354 INFO [archive_tarball] publishing log messages to tcp://*:5565
2024-04-24 08:56:05,355 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-24 08:56:05,356 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-aug20.tar from /ocean/sallen/timbucktwo/oxygen/*aug20/
2024-04-24 09:03:11,076 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-aug20.index from /ocean/dlatorne/oxygen-aug20.tar
2024-04-24 09:03:12,606 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-aug20.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-24 09:15:33,935 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240424/12/
2024-04-24 09:15:33,936 INFO [collect_weather] 2024-04-24 2.5km weather forecast 12 collection complete
2024-04-24 09:15:33,939 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-24 09:15:34,248 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-04-23'], host='localhost')
2024-04-24 09:15:34,253 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-04-23'], host='localhost')
2024-04-24 09:15:34,257 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-04-23'], host='localhost')
2024-04-24 09:15:34,261 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-04-23'], host='localhost')
2024-04-24 09:15:34,265 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-04-24 09:15:34,271 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-04-24 09:15:34,286 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-04-24 09:15:34,293 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-04-24 09:15:34,297 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-04-24 09:15:36,193 INFO [collect_NeahBay_ssh] running in process 2130594
2024-04-24 09:15:36,221 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:36,231 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-04-24 09:15:36,235 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:36,236 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-04-24
2024-04-24 09:15:36,241 INFO [collect_weather] running in process 2130596
2024-04-24 09:15:36,245 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:36,248 INFO [collect_weather] publishing log messages to tcp://*:5566
2024-04-24 09:15:36,249 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:36,279 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-04-24 09:15:36,500 INFO [download_live_ocean] running in process 2130595
2024-04-24 09:15:36,513 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:36,513 INFO [download_live_ocean] publishing log messages to tcp://*:5567
2024-04-24 09:15:36,514 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:36,515 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-04-24
2024-04-24 09:15:38,067 INFO [collect_NeahBay_ssh] 2024-04-24T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-04-24 09:15:38,071 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-24 09:15:38,085 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 09:15:38,233 INFO [download_live_ocean] downloaded 10129543 bytes from https://liveocean.apl.uw.edu/output/f2024.04.24/ubc.nc to /results/forcing/LiveOcean/downloaded/20240424/low_passed_UBC.nc
2024-04-24 09:15:38,235 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-04-24 09:15:38,504 INFO [collect_river_data] running in process 2130591
2024-04-24 09:15:38,505 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:38,507 INFO [collect_river_data] publishing log messages to tcp://*:5564
2024-04-24 09:15:38,508 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:38,509 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-04-23
2024-04-24 09:15:38,578 INFO [collect_river_data] running in process 2130588
2024-04-24 09:15:38,580 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:38,585 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-04-24 09:15:38,586 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:38,587 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-04-23
2024-04-24 09:15:38,597 INFO [collect_river_data] running in process 2130592
2024-04-24 09:15:38,598 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:38,600 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-04-24 09:15:38,601 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:38,602 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-04-23
2024-04-24 09:15:38,607 INFO [collect_river_data] running in process 2130590
2024-04-24 09:15:38,608 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:38,609 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-24 09:15:38,614 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:38,615 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-04-23
2024-04-24 09:15:38,769 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-04-23&endDT=2024-04-23 "HTTP/1.1 200 OK"
2024-04-24 09:15:38,781 INFO [make_turbidity_file] running in process 2130593
2024-04-24 09:15:38,782 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:38,783 INFO [make_turbidity_file] publishing log messages to tcp://*:5571
2024-04-24 09:15:38,785 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:38,787 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-04-24
2024-04-24 09:15:38,788 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-04-23&endDT=2024-04-23 "HTTP/1.1 200 OK"
2024-04-24 09:15:38,923 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-04-23&endDT=2024-04-23 "HTTP/1.1 200 OK"
2024-04-24 09:15:38,951 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-04-23&endDT=2024-04-23 "HTTP/1.1 200 OK"
2024-04-24 09:15:39,280 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-04-23 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-04-24 09:15:39,281 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-04-23 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-04-24 09:15:39,281 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-04-23 completed
2024-04-24 09:15:39,282 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-04-23 completed
2024-04-24 09:15:39,285 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-24 09:15:39,298 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-04-23 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-04-24 09:15:39,299 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-04-23 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-04-24 09:15:39,300 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-04-23 completed
2024-04-24 09:15:39,301 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-04-23 completed
2024-04-24 09:15:39,303 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-24 09:15:39,314 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240424/low_passed_UBC.nc
2024-04-24 09:15:39,315 INFO [download_live_ocean] 2024-04-24 Live Ocean file for Salish Sea western boundary downloaded
2024-04-24 09:15:39,378 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-24 09:15:39,384 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-24 09:15:39,398 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-04-24 09:15:39,603 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-04-24'], host='localhost')
2024-04-24 09:15:39,760 INFO [crop_gribs] running in process 2130597
2024-04-24 09:15:39,764 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:39,765 INFO [crop_gribs] publishing log messages to tcp://*:5564
2024-04-24 09:15:39,766 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:39,769 INFO [crop_gribs] cropping 2024-04-24 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-04-24 09:15:39,774 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240424/18/
2024-04-24 09:15:42,215 INFO [make_turbidity_file] 2024-04-24 Fraser River turbidity file creation complete
2024-04-24 09:15:42,217 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-04-24 09:15:42,994 INFO [make_live_ocean_files] running in process 2130759
2024-04-24 09:15:42,995 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:42,996 INFO [make_live_ocean_files] publishing log messages to tcp://*:5567
2024-04-24 09:15:42,997 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:42,998 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-04-24 Live Ocean run
2024-04-24 09:15:43,065 INFO [make_ssh_files] running in process 2130725
2024-04-24 09:15:43,081 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:43,099 INFO [make_ssh_files] publishing log messages to tcp://*:5568
2024-04-24 09:15:43,100 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:43,101 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-04-24 NOAA Neah Bay 06Z observation and forecast values
2024-04-24 09:15:48,588 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d20.nc
2024-04-24 09:15:48,782 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d21.nc
2024-04-24 09:15:48,990 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d22.nc
2024-04-24 09:15:49,211 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d23.nc
2024-04-24 09:15:49,451 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d24.nc
2024-04-24 09:15:49,648 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d25.nc
2024-04-24 09:15:49,867 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d26.nc
2024-04-24 09:15:50,070 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d27.nc
2024-04-24 09:15:50,291 INFO [make_ssh_files] sea surface height boundary file for 2024-04-24 nowcast run created
2024-04-24 09:15:50,310 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-24 09:15:50,313 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-24 09:15:50,318 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-24 09:15:51,684 INFO [make_runoff_file] running in process 2130809
2024-04-24 09:15:51,686 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:51,687 INFO [make_runoff_file] publishing log messages to tcp://*:5568
2024-04-24 09:15:51,688 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:51,732 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-24 09:15:51,862 INFO [make_runoff_file] 2024-04-24 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-24 09:15:51,865 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-24 09:15:52,561 INFO [make_v202111_runoff_file] running in process 2130807
2024-04-24 09:15:52,562 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:15:52,563 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5568
2024-04-24 09:15:52,564 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:15:52,564 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-23
2024-04-24 09:15:55,986 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-23: /results/forcing/rivers/R202108Dailies_y2024m04d23.nc
2024-04-24 09:15:55,986 INFO [make_v202111_runoff_file] 2024-04-23 runoff file creation completed
2024-04-24 09:15:55,989 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-24 09:16:15,651 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240424/12/
2024-04-24 09:16:15,651 INFO [crop_gribs] 2024-04-24 12 GRIBs cropping complete
2024-04-24 09:16:15,654 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-24 09:16:15,848 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-04-24 09:16:17,879 INFO [grib_to_netcdf] running in process 2130853
2024-04-24 09:16:17,880 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:16:17,881 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2024-04-24 09:16:17,881 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:16:17,882 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-24 nowcast and forecast runs
2024-04-24 09:16:27,453 INFO [archive_tarball] hindcast *aug20 results files archived to robot.graham
2024-04-24 09:16:27,456 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-24 09:16:31,863 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2024-04-24 09:16:31,863 INFO [make_live_ocean_files] 2024-04-24 Live Ocean western boundary conditions files created
2024-04-24 09:16:31,865 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-04-24 09:16:34,634 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m04d24.nc
2024-04-24 09:16:49,028 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d25.nc
2024-04-24 09:16:55,388 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d26.nc
2024-04-24 09:16:55,407 INFO [grib_to_netcdf] 2024-04-24 NEMO-atmos forcing file for nowcast+ created
2024-04-24 09:16:55,443 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-24 09:16:55,472 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-04-24 09:16:55,477 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-04-24 09:16:55,481 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-04-24 09:16:55,486 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-04-24 09:16:55,490 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-04-24 09:16:56,407 INFO [ping_erddap] running in process 2130897
2024-04-24 09:16:56,408 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:16:56,409 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-24 09:16:56,410 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:16:56,417 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-04-24 09:16:56,419 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-24 09:16:56,671 INFO [upload_forcing] running in process 2130896
2024-04-24 09:16:56,672 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:16:56,673 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-04-24 09:16:56,674 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:16:56,688 INFO [upload_forcing] running in process 2130894
2024-04-24 09:16:56,689 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:16:56,690 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-24 09:16:56,691 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:16:56,718 INFO [upload_forcing] running in process 2130895
2024-04-24 09:16:56,720 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:16:56,720 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-04-24 09:16:56,721 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:16:56,733 INFO [upload_forcing] running in process 2130892
2024-04-24 09:16:56,735 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:16:56,736 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-24 09:16:56,736 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:16:59,074 INFO [upload_forcing] nowcast+ 2024-04-24 forcing files upload to orcinus-nowcast-agrif completed
2024-04-24 09:16:59,077 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-24 09:16:59,222 INFO [upload_forcing] nowcast+ 2024-04-24 forcing files upload to optimum-hindcast completed
2024-04-24 09:16:59,226 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-24 09:17:07,090 INFO [upload_forcing] nowcast+ 2024-04-24 forcing files upload to robot.graham completed
2024-04-24 09:17:07,111 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-24 09:17:11,275 INFO [upload_forcing] nowcast+ 2024-04-24 forcing files upload to arbutus.cloud-nowcast completed
2024-04-24 09:17:11,302 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-24 09:17:11,308 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 09:17:12,466 INFO [make_forcing_links] running in process 2130936
2024-04-24 09:17:12,468 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:17:12,469 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-04-24 09:17:12,470 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:17:13,637 INFO [make_forcing_links] nowcast+ 2024-04-24 forcing file links on arbutus.cloud-nowcast created
2024-04-24 09:17:13,641 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-24 09:17:13,658 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-24'], host='arbutus.cloud-nowcast')
2024-04-24 09:17:15,736 INFO [run_NEMO] running in process 32481
2024-04-24 09:17:15,737 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:17:15,738 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-24 09:17:15,859 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:17:16,610 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/24apr24nowcast_2024-04-24T091715.792190-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-24 09:17:16,633 INFO [run_NEMO] nowcast NEMO run for 2024-04-24 on arbutus.cloud-nowcast started
2024-04-24 09:17:16,637 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-24 09:17:16,654 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-24 09:17:18,639 INFO [watch_NEMO] running in process 32733
2024-04-24 09:17:18,649 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:17:18,649 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-24 09:17:18,650 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:17:18,710 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-24 09:22:18,711 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17608759 = 2024-04-24 04:52:00 UTC, 20.3% complete
2024-04-24 09:27:18,712 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17609234 = 2024-04-24 10:08:40 UTC, 42.3% complete
2024-04-24 09:32:18,712 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17609713 = 2024-04-24 15:28:00 UTC, 64.5% complete
2024-04-24 09:37:18,713 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17610201 = 2024-04-24 20:53:20 UTC, 87.1% complete
2024-04-24 09:42:18,717 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-04-24 09:42:18,720 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-24 09:42:18,935 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 09:42:18,940 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 09:42:20,267 INFO [make_forcing_links] running in process 2131155
2024-04-24 09:42:20,268 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:42:20,269 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-04-24 09:42:20,269 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:42:20,274 INFO [download_results] running in process 2131156
2024-04-24 09:42:20,275 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:42:20,276 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-24 09:42:20,277 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:42:20,278 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/24apr24 to /results/SalishSea/nowcast-blue.202111
2024-04-24 09:42:21,308 INFO [make_forcing_links] ssh 2024-04-24 forcing file links on arbutus.cloud-nowcast created
2024-04-24 09:42:21,312 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-24 09:42:21,332 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-24'], host='arbutus.cloud-nowcast')
2024-04-24 09:42:23,620 INFO [run_NEMO] running in process 663
2024-04-24 09:42:23,621 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:42:23,622 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-24 09:42:23,623 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:42:24,566 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/24apr24forecast_2024-04-24T094223.699002-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-24 09:42:24,579 INFO [run_NEMO] forecast NEMO run for 2024-04-24 on arbutus.cloud-nowcast started
2024-04-24 09:42:24,583 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-24 09:42:24,598 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-24 09:42:27,023 INFO [watch_NEMO] running in process 921
2024-04-24 09:42:27,024 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 09:42:27,025 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-24 09:42:27,026 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-24 09:42:27,156 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-24 09:47:27,707 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17610907 = 2024-04-25 04:44:00 UTC, 13.2% complete
2024-04-24 09:52:27,777 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17611392 = 2024-04-25 10:07:20 UTC, 28.1% complete
2024-04-24 09:57:27,816 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17611844 = 2024-04-25 15:08:40 UTC, 42.1% complete
2024-04-24 10:02:27,146 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17612294 = 2024-04-25 20:08:40 UTC, 56.0% complete
2024-04-24 10:02:27,411 INFO [download_results] nowcast 2024-04-24 results files from arbutus.cloud-nowcast downloaded
2024-04-24 10:02:27,478 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 10:02:27,480 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:02:27,486 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-04-23'], host='localhost')
2024-04-24 10:02:27,492 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:02:31,874 INFO [make_CHS_currents_file] running in process 2131278
2024-04-24 10:02:31,875 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:02:31,876 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-04-24 10:02:31,877 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:02:38,475 INFO [make_plots] running in process 2131277
2024-04-24 10:02:38,527 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:02:38,574 INFO [make_plots] publishing log messages to tcp://*:5565
2024-04-24 10:02:38,616 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:02:38,618 INFO [make_plots] running in process 2131275
2024-04-24 10:02:38,620 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:02:38,621 INFO [make_plots] publishing log messages to tcp://*:5567
2024-04-24 10:02:38,656 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:02:40,157 INFO [make_plots] preparing render list for 2024-04-23 NEMO nowcast-blue comparison figures
2024-04-24 10:02:40,159 INFO [make_plots] preparing render list for 2024-04-24 NEMO nowcast-blue research figures
2024-04-24 10:02:41,307 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-blue research render list
2024-04-24 10:02:42,153 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-blue research render list
2024-04-24 10:02:42,155 INFO [make_plots] added Currents_sections_and_surface figure to 2024-04-24 NEMO nowcast-blue research render list
2024-04-24 10:02:42,156 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-04-24 NEMO nowcast-blue research render list
2024-04-24 10:02:42,157 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-04-24 NEMO nowcast-blue research render list
2024-04-24 10:02:42,158 INFO [make_plots] starting to render nemo nowcast research 24apr24 figures
2024-04-24 10:03:08,495 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-24 for nowcast
2024-04-24 10:03:08,498 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-24 10:03:21,367 INFO [make_plots] added SH_wind figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,368 INFO [make_plots] added Compare_VENUS_East figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,369 INFO [make_plots] added Compare_VENUS_Central figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,370 INFO [make_plots] added Central_ADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,371 INFO [make_plots] added Central_depavADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,372 INFO [make_plots] added Central_timeavADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,373 INFO [make_plots] added East_ADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,374 INFO [make_plots] added East_depavADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,375 INFO [make_plots] added East_timeavADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,376 INFO [make_plots] added ddl_ADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,377 INFO [make_plots] added ddl_depavADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,378 INFO [make_plots] added ddl_timeavADCP figure to 2024-04-23 NEMO nowcast-blue comparison render list
2024-04-24 10:03:21,379 INFO [make_plots] starting to render nemo nowcast comparison 23apr24 figures
2024-04-24 10:04:07,022 INFO [make_plots] finished rendering nemo nowcast research 24apr24 figures
2024-04-24 10:04:07,025 INFO [make_plots] nemo research plots for 2024-04-24 nowcast completed
2024-04-24 10:04:07,044 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-24 10:05:24,715 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-24 10:05:25,178 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-24 10:05:25,637 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-24 10:05:26,101 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-24 10:05:26,564 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-24 10:05:27,026 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-24 10:05:27,486 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-24 10:05:27,953 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-24 10:05:28,415 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-24 10:05:28,416 INFO [make_plots] finished rendering nemo nowcast comparison 23apr24 figures
2024-04-24 10:05:28,511 INFO [make_plots] nemo comparison plots for 2024-04-23 nowcast completed
2024-04-24 10:05:28,515 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-24 10:07:27,148 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17612775 = 2024-04-26 01:29:20 UTC, 70.8% complete
2024-04-24 10:12:27,150 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17613244 = 2024-04-26 06:42:00 UTC, 85.3% complete
2024-04-24 10:17:27,152 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17613710 = 2024-04-26 11:52:40 UTC, 99.7% complete
2024-04-24 10:20:01,609 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 10:20:01,886 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-09-01'], host='localhost')
2024-04-24 10:20:01,892 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 10:20:01,897 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 10:20:03,711 INFO [download_results] running in process 2131469
2024-04-24 10:20:03,716 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:20:03,717 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-24 10:20:03,719 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:20:03,720 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01sep20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 10:20:07,862 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 10:21:24,439 INFO [download_results] hindcast 2020-09-01 results files from optimum-hindcast downloaded
2024-04-24 10:21:24,453 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 10:21:24,470 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-09-01'], host='localhost')
2024-04-24 10:21:26,106 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 10:22:27,157 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-04-24 10:22:27,196 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-24 10:22:27,526 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-24'], host='arbutus.cloud-nowcast')
2024-04-24 10:22:27,568 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-24'], host='arbutus.cloud-nowcast')
2024-04-24 10:22:27,570 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:22:27,574 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:22:27,579 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:22:27,583 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:22:27,587 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:22:29,116 INFO [upload_forcing] running in process 2131520
2024-04-24 10:22:29,120 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:29,121 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-04-24 10:22:29,122 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:29,138 INFO [upload_forcing] running in process 2131518
2024-04-24 10:22:29,139 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:29,140 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-04-24 10:22:29,141 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:29,143 INFO [download_results] running in process 2131522
2024-04-24 10:22:29,144 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:29,145 INFO [download_results] publishing log messages to tcp://*:5567
2024-04-24 10:22:29,146 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:29,147 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/24apr24 to /results/SalishSea/forecast.202111
2024-04-24 10:22:29,149 INFO [upload_forcing] running in process 2131519
2024-04-24 10:22:29,150 INFO [upload_forcing] running in process 2131521
2024-04-24 10:22:29,151 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:29,153 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:29,154 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-24 10:22:29,155 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-04-24 10:22:29,156 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:29,157 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:29,921 INFO [upload_forcing] turbidity 2024-04-24 forcing files upload to optimum-hindcast completed
2024-04-24 10:22:29,926 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-24 10:22:29,961 INFO [upload_forcing] turbidity 2024-04-24 forcing files upload to orcinus-nowcast-agrif completed
2024-04-24 10:22:29,965 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-24 10:22:29,986 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:22:30,230 INFO [upload_forcing] turbidity 2024-04-24 forcing files upload to arbutus.cloud-nowcast completed
2024-04-24 10:22:30,290 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-24 10:22:30,292 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:22:31,371 INFO [make_forcing_links] running in process 2131553
2024-04-24 10:22:31,382 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:31,383 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-04-24 10:22:31,384 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:31,750 INFO [make_forcing_links] running in process 2131555
2024-04-24 10:22:31,770 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:31,771 INFO [make_forcing_links] publishing log messages to tcp://*:5568
2024-04-24 10:22:31,772 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:32,224 INFO [make_forcing_links] nowcast-agrif 2024-04-24 forcing file links on orcinus-nowcast-agrif created
2024-04-24 10:22:32,229 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-24 10:22:32,250 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:22:32,783 INFO [upload_forcing] turbidity 2024-04-24 forcing files upload to robot.graham completed
2024-04-24 10:22:32,787 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-24 10:22:33,723 INFO [run_NEMO_agrif] running in process 2131569
2024-04-24 10:22:33,724 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:33,725 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5562
2024-04-24 10:22:33,726 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:34,301 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-24 10:22:36,435 INFO [make_forcing_links] nowcast-green 2024-04-24 forcing file links on arbutus.cloud-nowcast created
2024-04-24 10:22:36,437 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-24 10:22:36,459 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-24'], host='arbutus.cloud-nowcast')
2024-04-24 10:22:40,100 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/24apr24nowcast-green_2024-04-24T102239.179386-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-24 10:22:40,113 INFO [run_NEMO] nowcast-green NEMO run for 2024-04-24 on arbutus.cloud-nowcast started
2024-04-24 10:22:40,116 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-24 10:22:40,157 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-04-24 10:22:43,112 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-24 10:22:48,657 INFO [run_NEMO_agrif] job id for 24apr24nowcast-agrif: 11093034.orca2.ibb
2024-04-24 10:22:48,669 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-04-24 queued on orcinus-nowcast-agrif
2024-04-24 10:22:48,671 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-04-24 10:22:48,691 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11093034.orca2.ibb'], host='localhost')
2024-04-24 10:22:49,982 INFO [watch_NEMO_agrif] running in process 2131587
2024-04-24 10:22:49,984 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:22:49,985 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5562
2024-04-24 10:22:49,986 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:22:50,801 INFO [watch_NEMO_agrif] watching 24apr24nowcast-agrif job 11093034 on orcinus-nowcast-agrif
2024-04-24 10:22:51,040 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif job 11093034 is queued on orcinus-nowcast-agrif
2024-04-24 10:23:14,314 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-24 10:23:14,367 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-24'], host='arbutus.cloud-nowcast')
2024-04-24 10:23:25,177 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-24 10:23:25,202 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-24 10:23:36,912 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-24 10:23:51,255 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif job 11093034 is queued on orcinus-nowcast-agrif
2024-04-24 10:24:51,470 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif job 11093034 is queued on orcinus-nowcast-agrif
2024-04-24 10:25:52,407 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-04-24 10:27:43,070 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17608477 = 2024-04-24 01:44:00 UTC, 7.2% complete
2024-04-24 10:28:36,951 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2024-04-24 10:30:52,856 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7629308 = 2024-04-24 02:04:40 UTC, 8.7% complete
2024-04-24 10:32:43,523 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17608652 = 2024-04-24 03:40:40 UTC, 15.3% complete
2024-04-24 10:33:36,911 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete
2024-04-24 10:35:53,311 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7629506 = 2024-04-24 04:16:40 UTC, 17.8% complete
2024-04-24 10:37:43,699 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17608826 = 2024-04-24 05:36:40 UTC, 23.4% complete
2024-04-24 10:38:36,926 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 291, 67.4% complete
2024-04-24 10:40:53,801 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7629706 = 2024-04-24 06:30:00 UTC, 27.1% complete
2024-04-24 10:42:43,587 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17609000 = 2024-04-24 07:32:40 UTC, 31.4% complete
2024-04-24 10:43:36,695 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 387, 89.6% complete
2024-04-24 10:45:54,249 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7629907 = 2024-04-24 08:44:00 UTC, 36.4% complete
2024-04-24 10:47:43,508 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17609176 = 2024-04-24 09:30:00 UTC, 39.6% complete
2024-04-24 10:48:36,834 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-24 10:48:36,836 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-24 10:48:37,177 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:48:37,183 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-24'], host='arbutus.cloud-nowcast')
2024-04-24 10:48:38,901 INFO [download_wwatch3_results] running in process 2131725
2024-04-24 10:48:38,903 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:48:38,904 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-04-24 10:48:38,904 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:48:45,566 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-24 10:48:45,606 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-24 10:48:55,543 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-24 10:50:54,730 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7630107 = 2024-04-24 10:57:20 UTC, 45.7% complete
2024-04-24 10:52:44,347 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17609350 = 2024-04-24 11:26:00 UTC, 47.7% complete
2024-04-24 10:53:45,997 INFO [download_wwatch3_results] nowcast 2024-04-24 results files from arbutus.cloud-nowcast downloaded
2024-04-24 10:53:46,001 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-24 10:53:55,515 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete
2024-04-24 10:54:06,841 INFO [download_results] forecast 2024-04-24 results files from arbutus.cloud-nowcast downloaded
2024-04-24 10:54:06,846 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 10:54:06,870 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:54:11,339 INFO [make_CHS_currents_file] running in process 2131741
2024-04-24 10:54:11,340 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:54:11,341 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-04-24 10:54:11,343 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:54:58,887 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-24 for forecast
2024-04-24 10:54:58,890 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-24 10:54:58,913 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:54:59,964 INFO [update_forecast_datasets] running in process 2131766
2024-04-24 10:54:59,997 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:54:59,998 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-04-24 10:54:59,999 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:54:59,999 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/24apr24 run
2024-04-24 10:55:00,525 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/24apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-24 10:55:00,528 INFO [update_forecast_datasets] nemo 2024-04-24 forecast rolling forecast datasets updated
2024-04-24 10:55:00,530 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-24 10:55:00,555 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-24 10:55:00,559 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:55:00,564 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 10:55:01,919 INFO [ping_erddap] running in process 2131771
2024-04-24 10:55:01,920 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:55:01,921 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-04-24 10:55:01,922 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:55:01,934 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-24 10:55:01,936 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-24 10:55:04,336 INFO [make_surface_current_tiles] running in process 2131774
2024-04-24 10:55:04,337 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:55:04,338 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-04-24 10:55:04,339 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:55:04,377 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-24 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/24apr24
2024-04-24 10:55:09,982 INFO [make_plots] running in process 2131773
2024-04-24 10:55:09,987 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 10:55:09,988 INFO [make_plots] publishing log messages to tcp://*:5567
2024-04-24 10:55:09,989 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-24 10:55:11,322 INFO [make_plots] preparing render list for 2024-04-24 NEMO forecast publish figures
2024-04-24 10:55:11,461 INFO [make_plots] added Website_thumbnail figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,465 INFO [make_plots] added Threshold_website figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,465 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,466 INFO [make_plots] added Boundary Bay figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,478 INFO [make_plots] added Campbell River figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,480 INFO [make_plots] added Cherry Point figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,481 INFO [make_plots] added Friday Harbor figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,482 INFO [make_plots] added Halfmoon Bay figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,483 INFO [make_plots] added Nanaimo figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,484 INFO [make_plots] added Neah Bay figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,485 INFO [make_plots] added New Westminster figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,486 INFO [make_plots] added Patricia Bay figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,487 INFO [make_plots] added Point Atkinson figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,488 INFO [make_plots] added Port Renfrew figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,489 INFO [make_plots] added Sand Heads figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,490 INFO [make_plots] added Sandy Cove figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,490 INFO [make_plots] added Squamish figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,491 INFO [make_plots] added Victoria figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,492 INFO [make_plots] added Woodwards Landing figure to 2024-04-24 NEMO forecast publish render list
2024-04-24 10:55:11,493 INFO [make_plots] starting to render nemo forecast publish 24apr24 figures
2024-04-24 10:55:55,174 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7630305 = 2024-04-24 13:09:20 UTC, 54.8% complete
2024-04-24 10:57:43,054 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17609523 = 2024-04-24 13:21:20 UTC, 55.7% complete
2024-04-24 10:58:54,871 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 183, 28.2% complete
2024-04-24 11:00:55,655 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7630505 = 2024-04-24 15:22:40 UTC, 64.1% complete
2024-04-24 11:02:43,048 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17609697 = 2024-04-24 15:17:20 UTC, 63.7% complete
2024-04-24 11:03:54,871 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 282, 43.5% complete
2024-04-24 11:05:56,142 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7630705 = 2024-04-24 17:36:00 UTC, 73.4% complete
2024-04-24 11:07:43,049 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17609872 = 2024-04-24 17:14:00 UTC, 71.8% complete
2024-04-24 11:08:54,872 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 381, 58.8% complete
2024-04-24 11:09:48,164 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/24apr24
2024-04-24 11:10:56,593 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7630905 = 2024-04-24 19:49:20 UTC, 82.6% complete
2024-04-24 11:12:43,052 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17610046 = 2024-04-24 19:10:00 UTC, 79.9% complete
2024-04-24 11:13:54,872 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 480, 74.1% complete
2024-04-24 11:13:59,700 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-24 11:13:59,719 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-24 11:14:25,300 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/24apr24
2024-04-24 11:14:25,317 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-24 forecast into /results/nowcast-sys/figures/surface_currents/forecast/24apr24
2024-04-24 11:14:25,318 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-24 forecast completed
2024-04-24 11:14:25,668 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-24 11:14:38,309 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:15:57,048 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7631103 = 2024-04-24 22:01:20 UTC, 91.8% complete
2024-04-24 11:16:32,783 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:16:33,190 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:17:11,160 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:17:43,055 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17610220 = 2024-04-24 21:06:00 UTC, 88.0% complete
2024-04-24 11:18:27,756 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:18:54,873 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 579, 89.4% complete
2024-04-24 11:19:06,938 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:19:44,749 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:20:24,204 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:20:57,582 INFO [watch_NEMO_agrif] 24apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7631280 = 2024-04-24 23:59:20 UTC, 100.0% complete
2024-04-24 11:21:02,165 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:21:40,784 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:22:19,620 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:22:25,091 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:22:43,052 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17610394 = 2024-04-24 23:02:00 UTC, 96.0% complete
2024-04-24 11:23:02,935 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:23:41,079 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-19 00:00:00Z to 2024-04-26 11:55:00Z
2024-04-24 11:23:54,874 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-24 11:24:19,229 INFO [make_plots] finished rendering nemo forecast publish 24apr24 figures
2024-04-24 11:24:19,232 INFO [make_plots] nemo publish plots for 2024-04-24 forecast completed
2024-04-24 11:24:19,240 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-24 11:24:19,411 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 11:24:24,817 INFO [make_feeds] running in process 2134440
2024-04-24 11:24:24,818 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:24:24,819 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-04-24 11:24:24,820 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:24:53,916 INFO [make_feeds] ATOM feeds for 2024-04-24 forecast run completed
2024-04-24 11:24:53,981 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-24 11:25:57,823 INFO [watch_NEMO_agrif] NEMO AGRIF run 11093034.orca2.ibb on orcinus-nowcast-agrif completed
2024-04-24 11:25:57,828 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-04-24 11:25:58,203 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 11:25:59,798 INFO [download_results] running in process 2134466
2024-04-24 11:25:59,805 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:25:59,806 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-24 11:25:59,807 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:25:59,808 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/24apr24 to /results/SalishSea/nowcast-agrif.201702
2024-04-24 11:26:29,481 INFO [download_results] nowcast-agrif 2024-04-24 results files from orcinus-nowcast-agrif downloaded
2024-04-24 11:26:29,487 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 11:26:29,679 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 11:26:34,667 INFO [make_plots] running in process 2134473
2024-04-24 11:26:34,668 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:26:34,670 INFO [make_plots] publishing log messages to tcp://*:5562
2024-04-24 11:26:34,671 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:26:35,821 INFO [make_plots] preparing render list for 2024-04-24 NEMO nowcast-agrif research figures
2024-04-24 11:26:35,851 INFO [make_plots] added baynes_sound_surface figure to 2024-04-24 NEMO nowcast-agrif research render list
2024-04-24 11:26:35,853 INFO [make_plots] starting to render nemo nowcast-agrif research 24apr24 figures
2024-04-24 11:26:45,370 INFO [make_plots] finished rendering nemo nowcast-agrif research 24apr24 figures
2024-04-24 11:26:45,372 INFO [make_plots] nemo research plots for 2024-04-24 nowcast-agrif completed
2024-04-24 11:26:45,376 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-24 11:27:43,052 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17610480 = 2024-04-24 23:59:20 UTC, 100.0% complete
2024-04-24 11:28:54,876 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-24 11:28:54,914 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-24 11:28:55,284 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 11:28:56,228 INFO [download_wwatch3_results] running in process 2134515
2024-04-24 11:28:56,229 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:28:56,230 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2024-04-24 11:28:56,232 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:31:30,121 INFO [download_wwatch3_results] forecast 2024-04-24 results files from arbutus.cloud-nowcast downloaded
2024-04-24 11:31:30,125 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-24 11:31:30,308 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 11:31:31,405 INFO [update_forecast_datasets] running in process 2134527
2024-04-24 11:31:31,444 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:31:31,445 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-04-24 11:31:31,446 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:31:31,447 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/24apr24 run
2024-04-24 11:31:31,473 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/24apr24 run
2024-04-24 11:31:31,593 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/24apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-24 11:31:31,594 INFO [update_forecast_datasets] wwatch3 2024-04-24 forecast rolling forecast datasets updated
2024-04-24 11:31:31,596 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-24 11:31:31,794 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-24 11:31:32,704 INFO [ping_erddap] running in process 2134532
2024-04-24 11:31:32,705 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:31:32,706 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-24 11:31:32,707 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:31:32,708 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-24 11:31:32,710 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-24 11:31:32,891 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 11:31:37,344 INFO [make_plots] running in process 2134537
2024-04-24 11:31:37,346 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:31:37,347 INFO [make_plots] publishing log messages to tcp://*:5562
2024-04-24 11:31:37,348 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:31:37,349 INFO [make_plots] preparing render list for 2024-04-24 WaveWatch3 forecast publish figures
2024-04-24 11:31:37,350 INFO [make_plots] added Halibut Bank figure to 2024-04-24 WaveWatch3 forecast publish render list
2024-04-24 11:31:37,351 INFO [make_plots] added Sentry Shoal figure to 2024-04-24 WaveWatch3 forecast publish render list
2024-04-24 11:31:37,352 INFO [make_plots] starting to render wwatch3 forecast publish 24apr24 figures
2024-04-24 11:31:37,735 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-24 11:31:51,312 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-24 11:31:57,938 INFO [make_plots] finished rendering wwatch3 forecast publish 24apr24 figures
2024-04-24 11:31:57,974 INFO [make_plots] wwatch3 publish plots for 2024-04-24 forecast completed
2024-04-24 11:31:58,035 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-24 11:32:43,077 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-24 11:32:43,086 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-04-24 11:32:43,455 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 11:32:45,009 INFO [download_results] running in process 2134580
2024-04-24 11:32:45,010 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:32:45,012 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-24 11:32:45,013 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:32:45,014 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/24apr24 to /results2/SalishSea/nowcast-green.202111
2024-04-24 11:45:12,350 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 11:45:12,952 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-09-06'], host='localhost')
2024-04-24 11:45:12,959 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 11:45:12,965 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 11:45:15,428 INFO [download_results] running in process 2134603
2024-04-24 11:45:15,429 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 11:45:15,430 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-24 11:45:15,431 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 11:45:15,432 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06sep20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 11:45:19,420 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 11:46:35,012 INFO [download_results] hindcast 2020-09-06 results files from optimum-hindcast downloaded
2024-04-24 11:46:35,029 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 11:46:35,232 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-09-06'], host='localhost')
2024-04-24 11:46:37,018 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 12:06:48,785 INFO [download_results] nowcast-green 2024-04-24 results files from arbutus.cloud-nowcast downloaded
2024-04-24 12:06:48,789 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 12:06:48,979 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 12:06:48,985 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-04-24 12:06:48,991 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 12:06:48,996 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 12:06:49,001 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-04-24'], host='localhost')
2024-04-24 12:06:50,169 INFO [ping_erddap] running in process 2134694
2024-04-24 12:06:50,170 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 12:06:50,171 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-24 12:06:50,172 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-24 12:06:50,178 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-04-24 12:06:50,180 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-24 12:06:53,056 INFO [make_averaged_dataset] running in process 2134697
2024-04-24 12:06:53,057 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 12:06:53,058 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2024-04-24 12:06:53,059 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-24 12:06:53,060 INFO [make_averaged_dataset] creating day-averaged dataset for 24-Apr-2024 biology
2024-04-24 12:06:53,062 INFO [make_averaged_dataset] running in process 2134696
2024-04-24 12:06:53,063 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 12:06:53,064 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2024-04-24 12:06:53,065 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-24 12:06:53,066 INFO [make_averaged_dataset] creating day-averaged dataset for 24-Apr-2024 chemistry
2024-04-24 12:06:53,070 INFO [make_averaged_dataset] running in process 2134695
2024-04-24 12:06:53,071 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 12:06:53,072 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567
2024-04-24 12:06:53,073 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-24 12:06:53,073 INFO [make_averaged_dataset] creating day-averaged dataset for 24-Apr-2024 physics
2024-04-24 12:06:56,924 INFO [make_plots] running in process 2134692
2024-04-24 12:06:56,947 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 12:06:56,960 INFO [make_plots] publishing log messages to tcp://*:5568
2024-04-24 12:06:56,966 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-24 12:06:57,692 INFO [make_plots] preparing render list for 2024-04-24 NEMO nowcast-green research figures
2024-04-24 12:06:59,862 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:00,596 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:01,384 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:02,223 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:03,033 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:03,809 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:04,548 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:05,383 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:06,331 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:07,240 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:08,110 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:08,956 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:09,388 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:09,439 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:09,475 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:09,505 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-04-24 NEMO nowcast-green research render list
2024-04-24 12:07:09,510 INFO [make_plots] starting to render nemo nowcast-green research 24apr24 figures
2024-04-24 12:16:14,442 INFO [make_averaged_dataset] day-averaged dataset for 24-Apr-2024 physics created
2024-04-24 12:16:14,444 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-24 12:17:20,851 INFO [make_averaged_dataset] day-averaged dataset for 24-Apr-2024 chemistry created
2024-04-24 12:17:20,854 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-24 12:17:38,706 INFO [make_averaged_dataset] day-averaged dataset for 24-Apr-2024 biology created
2024-04-24 12:17:38,710 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-24 13:15:24,404 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 13:15:24,840 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-09-11'], host='localhost')
2024-04-24 13:15:24,850 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 13:15:24,854 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 13:15:27,886 INFO [download_results] running in process 2135367
2024-04-24 13:15:27,888 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 13:15:27,888 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-24 13:15:27,889 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 13:15:27,890 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11sep20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 13:15:31,853 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 13:17:00,374 INFO [download_results] hindcast 2020-09-11 results files from optimum-hindcast downloaded
2024-04-24 13:17:00,388 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 13:17:00,573 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-09-11'], host='localhost')
2024-04-24 13:17:01,724 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 14:08:17,505 INFO [make_plots] finished rendering nemo nowcast-green research 24apr24 figures
2024-04-24 14:08:17,712 INFO [make_plots] nemo research plots for 2024-04-24 nowcast-green completed
2024-04-24 14:08:17,793 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-24 14:40:36,688 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 14:40:37,123 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-09-16'], host='localhost')
2024-04-24 14:40:37,130 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 14:40:37,134 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 14:40:39,039 INFO [download_results] running in process 2135756
2024-04-24 14:40:39,041 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 14:40:39,042 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-24 14:40:39,043 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 14:40:39,044 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16sep20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 14:40:43,040 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 14:41:47,915 INFO [download_results] hindcast 2020-09-16 results files from optimum-hindcast downloaded
2024-04-24 14:41:47,931 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 14:41:48,152 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-09-16'], host='localhost')
2024-04-24 14:41:49,564 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 15:17:17,963 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240424/18/
2024-04-24 15:17:17,964 INFO [collect_weather] 2024-04-24 2.5km weather forecast 18 collection complete
2024-04-24 15:17:17,967 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-24 15:17:18,401 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-04-24 15:17:18,460 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-04-24 15:17:18,471 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-04-24 15:17:18,477 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-04-25'], host='localhost')
2024-04-24 15:17:20,004 INFO [download_weather] running in process 2135960
2024-04-24 15:17:20,005 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 15:17:20,006 INFO [download_weather] publishing log messages to tcp://*:5562
2024-04-24 15:17:20,007 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-24 15:17:20,008 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240424
2024-04-24 15:17:20,009 INFO [download_weather] running in process 2135958
2024-04-24 15:17:20,010 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 15:17:20,011 INFO [download_weather] publishing log messages to tcp://*:5563
2024-04-24 15:17:20,012 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-24 15:17:20,012 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240424
2024-04-24 15:17:20,030 INFO [collect_weather] running in process 2135961
2024-04-24 15:17:20,031 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 15:17:20,032 INFO [collect_weather] publishing log messages to tcp://*:5566
2024-04-24 15:17:20,033 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-24 15:17:20,133 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-04-24 15:17:22,031 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240424/18/
2024-04-24 15:17:22,032 INFO [crop_gribs] 2024-04-24 18 GRIBs cropping complete
2024-04-24 15:17:22,034 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-24 15:17:22,771 INFO [crop_gribs] running in process 2135962
2024-04-24 15:17:22,772 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 15:17:22,773 INFO [crop_gribs] publishing log messages to tcp://*:5564
2024-04-24 15:17:22,775 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-24 15:17:22,776 INFO [crop_gribs] cropping 2024-04-25 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-04-24 15:17:22,783 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240425/00/
2024-04-24 15:26:42,593 INFO [download_weather] 2024-04-24 1km weather forecast 12 downloads complete
2024-04-24 15:26:42,597 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-24 15:26:43,579 INFO [download_weather] 2024-04-24 1km weather forecast 00 downloads complete
2024-04-24 15:26:43,582 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-24 16:05:47,624 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 16:05:48,078 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-09-21'], host='localhost')
2024-04-24 16:05:48,093 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 16:05:48,114 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 16:05:49,917 INFO [download_results] running in process 2136131
2024-04-24 16:05:49,927 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 16:05:49,945 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-24 16:05:49,958 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 16:05:49,959 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21sep20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 16:05:54,126 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 16:06:50,791 INFO [download_results] hindcast 2020-09-21 results files from optimum-hindcast downloaded
2024-04-24 16:06:50,804 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 16:06:51,056 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-09-21'], host='localhost')
2024-04-24 16:06:52,055 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 17:30:58,693 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 17:30:59,112 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-09-26'], host='localhost')
2024-04-24 17:30:59,119 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 17:30:59,123 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 17:31:00,721 INFO [download_results] running in process 2136884
2024-04-24 17:31:00,722 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 17:31:00,723 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-24 17:31:00,724 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 17:31:00,725 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26sep20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 17:31:04,764 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 17:32:02,359 INFO [download_results] hindcast 2020-09-26 results files from optimum-hindcast downloaded
2024-04-24 17:32:02,374 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 17:32:02,592 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-09-26'], host='localhost')
2024-04-24 17:32:03,708 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 17:32:03,947 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2020-sep', 'robot.graham'], host='localhost')
2024-04-24 17:32:04,876 INFO [archive_tarball] running in process 2136927
2024-04-24 17:32:04,878 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 17:32:04,879 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-04-24 17:32:04,880 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-24 17:32:04,881 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-sep20.tar from /ocean/sallen/timbucktwo/oxygen/*sep20/
2024-04-24 17:39:15,009 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-sep20.index from /ocean/dlatorne/oxygen-sep20.tar
2024-04-24 17:39:15,108 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-sep20.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-24 18:20:49,864 INFO [archive_tarball] hindcast *sep20 results files archived to robot.graham
2024-04-24 18:20:49,901 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-24 18:56:09,358 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 18:56:09,774 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-10-01'], host='localhost')
2024-04-24 18:56:09,780 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 18:56:09,786 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 18:56:11,161 INFO [download_results] running in process 2137328
2024-04-24 18:56:11,162 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 18:56:11,163 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-24 18:56:11,164 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 18:56:11,166 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01oct20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 18:56:15,222 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 18:56:59,686 INFO [download_results] hindcast 2020-10-01 results files from optimum-hindcast downloaded
2024-04-24 18:56:59,700 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 18:56:59,911 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-10-01'], host='localhost')
2024-04-24 18:57:00,876 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 20:26:19,930 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 20:26:20,400 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-10-06'], host='localhost')
2024-04-24 20:26:20,406 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 20:26:20,413 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 20:26:22,153 INFO [download_results] running in process 2137959
2024-04-24 20:26:22,154 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 20:26:22,155 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-24 20:26:22,156 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 20:26:22,157 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06oct20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 20:26:26,283 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 20:27:24,711 INFO [download_results] hindcast 2020-10-06 results files from optimum-hindcast downloaded
2024-04-24 20:27:24,730 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 20:27:24,951 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-10-06'], host='localhost')
2024-04-24 20:27:26,220 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 21:05:13,663 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240425/00/
2024-04-24 21:05:13,664 INFO [collect_weather] 2024-04-25 2.5km weather forecast 00 collection complete
2024-04-24 21:05:13,667 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-24 21:05:14,102 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-04-24 21:05:14,108 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-04-25'], host='localhost')
2024-04-24 21:05:15,664 INFO [collect_weather] running in process 2138228
2024-04-24 21:05:15,671 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 21:05:15,672 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-04-24 21:05:15,674 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-24 21:05:15,684 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-04-24 21:05:19,120 INFO [crop_gribs] running in process 2138230
2024-04-24 21:05:19,121 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 21:05:19,122 INFO [crop_gribs] publishing log messages to tcp://*:5565
2024-04-24 21:05:19,123 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-24 21:05:19,124 INFO [crop_gribs] cropping 2024-04-25 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-04-24 21:05:19,136 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240425/06/
2024-04-24 21:13:12,587 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240425/00/
2024-04-24 21:13:12,588 INFO [crop_gribs] 2024-04-25 00 GRIBs cropping complete
2024-04-24 21:13:12,591 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-24 21:51:30,863 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 21:51:31,273 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-10-11'], host='localhost')
2024-04-24 21:51:31,293 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 21:51:31,297 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 21:51:33,472 INFO [download_results] running in process 2138371
2024-04-24 21:51:33,473 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 21:51:33,474 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-24 21:51:33,474 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 21:51:33,475 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11oct20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 21:51:37,688 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 21:52:34,891 INFO [download_results] hindcast 2020-10-11 results files from optimum-hindcast downloaded
2024-04-24 21:52:34,906 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 21:52:35,124 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-10-11'], host='localhost')
2024-04-24 21:52:36,268 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-24 23:16:42,059 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-24 23:16:42,463 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-10-16'], host='localhost')
2024-04-24 23:16:42,469 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 23:16:42,474 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-24 23:16:43,980 INFO [download_results] running in process 2138579
2024-04-24 23:16:43,981 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-24 23:16:43,982 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-24 23:16:43,983 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-24 23:16:43,983 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16oct20 to /ocean/sallen/timbucktwo/oxygen
2024-04-24 23:16:47,981 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-24 23:17:46,229 INFO [download_results] hindcast 2020-10-16 results files from optimum-hindcast downloaded
2024-04-24 23:17:46,245 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-24 23:17:46,463 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-10-16'], host='localhost')
2024-04-24 23:17:47,505 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-25 00:41:53,073 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-25 00:41:53,532 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-10-21'], host='localhost')
2024-04-25 00:41:53,543 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-25 00:41:53,549 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-25 00:41:55,558 INFO [download_results] running in process 2139560
2024-04-25 00:41:55,559 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 00:41:55,560 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-25 00:41:55,561 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-25 00:41:55,562 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21oct20 to /ocean/sallen/timbucktwo/oxygen
2024-04-25 00:41:59,816 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-25 00:42:57,770 INFO [download_results] hindcast 2020-10-21 results files from optimum-hindcast downloaded
2024-04-25 00:42:57,801 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-25 00:42:58,026 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-10-21'], host='localhost')
2024-04-25 00:42:59,259 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-25 02:27:06,311 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-25 02:27:06,720 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-10-26'], host='localhost')
2024-04-25 02:27:06,725 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-25 02:27:06,729 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-25 02:27:08,239 INFO [download_results] running in process 2139841
2024-04-25 02:27:08,241 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 02:27:08,242 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-25 02:27:08,243 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-25 02:27:08,243 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26oct20 to /ocean/sallen/timbucktwo/oxygen
2024-04-25 02:27:12,260 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-25 02:28:09,614 INFO [download_results] hindcast 2020-10-26 results files from optimum-hindcast downloaded
2024-04-25 02:28:09,631 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-25 02:28:09,858 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-10-26'], host='localhost')
2024-04-25 02:28:10,977 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-25 02:28:11,201 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2020-oct', 'robot.graham'], host='localhost')
2024-04-25 02:28:12,171 INFO [archive_tarball] running in process 2139890
2024-04-25 02:28:12,172 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 02:28:12,173 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-04-25 02:28:12,173 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-25 02:28:12,174 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct20.tar from /ocean/sallen/timbucktwo/oxygen/*oct20/
2024-04-25 02:35:29,581 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct20.index from /ocean/dlatorne/oxygen-oct20.tar
2024-04-25 02:35:45,323 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-oct20.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-25 03:13:37,451 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240425/06/
2024-04-25 03:13:37,452 INFO [collect_weather] 2024-04-25 2.5km weather forecast 06 collection complete
2024-04-25 03:13:37,455 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-25 03:13:37,896 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,904 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,908 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,916 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,923 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,931 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,935 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,940 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,944 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,947 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,969 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,978 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,985 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:37,994 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-04-24'], host='localhost')
2024-04-25 03:13:38,003 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-04-25 03:13:38,017 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-04-25 03:13:38,026 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-04-25 03:13:38,030 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-04-25 03:13:38,039 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-04-25 03:13:38,050 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-04-25 03:13:40,356 INFO [collect_weather] running in process 2140121
2024-04-25 03:13:40,357 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:40,358 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-04-25 03:13:40,359 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:40,376 INFO [collect_NeahBay_ssh] running in process 2140120
2024-04-25 03:13:40,377 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:40,377 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5566
2024-04-25 03:13:40,379 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:40,381 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-04-25
2024-04-25 03:13:40,384 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-04-25 03:13:42,292 INFO [collect_NeahBay_ssh] 2024-04-25T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-04-25 03:13:42,301 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-25 03:13:42,693 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-04-25'], host='localhost')
2024-04-25 03:13:42,695 INFO [collect_river_data] running in process 2140113
2024-04-25 03:13:42,696 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,697 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-04-25 03:13:42,698 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,699 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-04-24
2024-04-25 03:13:42,723 INFO [collect_river_data] running in process 2140104
2024-04-25 03:13:42,724 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,724 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-04-25 03:13:42,726 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,727 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-04-24
2024-04-25 03:13:42,806 INFO [collect_river_data] running in process 2140106
2024-04-25 03:13:42,807 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,807 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-04-25 03:13:42,808 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,809 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-04-24
2024-04-25 03:13:42,822 INFO [collect_river_data] running in process 2140107
2024-04-25 03:13:42,823 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,824 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-25 03:13:42,826 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,827 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-04-24
2024-04-25 03:13:42,836 INFO [collect_river_data] running in process 2140111
2024-04-25 03:13:42,837 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,838 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-04-25 03:13:42,839 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,840 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-04-24
2024-04-25 03:13:42,846 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-04-25 03:13:42,847 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-04-24 completed
2024-04-25 03:13:42,850 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:42,851 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-04-25 03:13:42,852 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-04-24 completed
2024-04-25 03:13:42,864 INFO [collect_river_data] running in process 2140116
2024-04-25 03:13:42,865 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,867 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-04-25 03:13:42,868 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,870 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-04-24
2024-04-25 03:13:42,872 INFO [collect_river_data] running in process 2140110
2024-04-25 03:13:42,873 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,876 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-04-25 03:13:42,877 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,878 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-04-24
2024-04-25 03:13:42,882 INFO [collect_river_data] running in process 2140108
2024-04-25 03:13:42,883 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,884 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-04-25 03:13:42,885 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,886 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-04-24
2024-04-25 03:13:42,892 INFO [collect_river_data] running in process 2140105
2024-04-25 03:13:42,893 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,895 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-04-25 03:13:42,897 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,898 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-04-24
2024-04-25 03:13:42,901 INFO [collect_river_data] running in process 2140102
2024-04-25 03:13:42,902 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,903 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-04-25 03:13:42,904 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,905 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-04-24
2024-04-25 03:13:42,909 INFO [collect_river_data] running in process 2140109
2024-04-25 03:13:42,910 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,910 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-04-25 03:13:42,912 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,914 INFO [collect_river_data] running in process 2140114
2024-04-25 03:13:42,915 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-04-24
2024-04-25 03:13:42,916 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,917 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-04-25 03:13:42,918 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,919 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-04-24
2024-04-25 03:13:42,920 INFO [collect_river_data] running in process 2140112
2024-04-25 03:13:42,922 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,923 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-04-25 03:13:42,924 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,935 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-04-24
2024-04-25 03:13:42,938 INFO [collect_river_data] running in process 2140115
2024-04-25 03:13:42,940 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:42,941 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-04-25 03:13:42,943 INFO [collect_river_data] publishing log messages to tcp://*:5590
2024-04-25 03:13:42,944 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-04-24 completed
2024-04-25 03:13:42,944 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:42,946 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-04-24
2024-04-25 03:13:42,948 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-04-24 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-04-25 03:13:42,949 INFO [collect_river_data] ECCC Fraser river data collection for 2024-04-24 completed
2024-04-25 03:13:42,979 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-04-25 03:13:42,980 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-04-24 completed
2024-04-25 03:13:42,998 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-04-25 03:13:42,999 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-04-24 completed
2024-04-25 03:13:43,006 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-04-25 03:13:43,008 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-04-24 completed
2024-04-25 03:13:43,024 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-04-25 03:13:43,025 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-04-24 completed
2024-04-25 03:13:43,029 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-04-25 03:13:43,030 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-04-24 completed
2024-04-25 03:13:43,034 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-04-25 03:13:43,037 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-04-24 completed
2024-04-25 03:13:43,039 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-04-25 03:13:43,040 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-04-24 completed
2024-04-25 03:13:43,068 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-04-24 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-04-25 03:13:43,069 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-04-24 completed
2024-04-25 03:13:43,105 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-04-24 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-04-25 03:13:43,106 INFO [collect_river_data] ECCC Capilano river data collection for 2024-04-24 completed
2024-04-25 03:13:43,214 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:43,352 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-04-24 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-04-25 03:13:43,353 INFO [collect_river_data] ECCC Englishman river data collection for 2024-04-24 completed
2024-04-25 03:13:43,464 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:43,545 INFO [crop_gribs] running in process 2140122
2024-04-25 03:13:43,546 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:43,547 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-04-25 03:13:43,548 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:43,549 INFO [crop_gribs] cropping 2024-04-25 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-04-25 03:13:43,558 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240425/12/
2024-04-25 03:13:43,703 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:43,931 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:44,086 INFO [get_onc_ferry] running in process 2140119
2024-04-25 03:13:44,088 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:44,089 INFO [get_onc_ferry] publishing log messages to tcp://*:5567
2024-04-25 03:13:44,090 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:44,090 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-04-24
2024-04-25 03:13:44,137 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:44,176 INFO [get_onc_ctd] running in process 2140118
2024-04-25 03:13:44,177 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:44,179 INFO [get_onc_ctd] publishing log messages to tcp://*:5568
2024-04-25 03:13:44,180 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:44,181 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-04-24
2024-04-25 03:13:44,195 INFO [get_onc_ctd] running in process 2140117
2024-04-25 03:13:44,196 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:44,197 INFO [get_onc_ctd] publishing log messages to tcp://*:5570
2024-04-25 03:13:44,198 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:44,198 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-04-24
2024-04-25 03:13:44,368 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:44,576 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:44,796 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:45,025 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:45,230 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:45,510 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:45,759 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:46,009 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-25 03:13:46,803 INFO [make_ssh_files] running in process 2140503
2024-04-25 03:13:46,828 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:46,836 INFO [make_ssh_files] publishing log messages to tcp://*:5569
2024-04-25 03:13:46,837 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:46,838 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-04-25 NOAA Neah Bay 00Z observation and forecast values
2024-04-25 03:13:52,496 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d20.nc
2024-04-25 03:13:52,706 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d21.nc
2024-04-25 03:13:52,899 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d22.nc
2024-04-25 03:13:53,114 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d23.nc
2024-04-25 03:13:53,331 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d24.nc
2024-04-25 03:13:53,791 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d25.nc
2024-04-25 03:13:53,995 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d26.nc
2024-04-25 03:13:54,226 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d27.nc
2024-04-25 03:13:54,428 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d28.nc
2024-04-25 03:13:54,679 INFO [make_ssh_files] sea surface height boundary file for 2024-04-25 forecast2 run created
2024-04-25 03:13:54,682 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-25 03:13:54,927 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-25 03:13:54,931 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-25 03:13:56,290 INFO [make_runoff_file] running in process 2140566
2024-04-25 03:13:56,293 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:56,294 INFO [make_runoff_file] publishing log messages to tcp://*:5569
2024-04-25 03:13:56,295 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:56,373 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-25 03:13:56,491 INFO [make_runoff_file] 2024-04-25 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-25 03:13:56,493 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-25 03:13:57,173 INFO [make_v202111_runoff_file] running in process 2140564
2024-04-25 03:13:57,174 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:13:57,175 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5569
2024-04-25 03:13:57,176 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:13:57,177 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-24
2024-04-25 03:13:58,336 ERROR [make_v202111_runoff_file] no 2024-04-24 discharge obs for Skagit_MountVernon: patching
2024-04-25 03:13:59,015 ERROR [make_v202111_runoff_file] no 2024-04-24 discharge obs for Snohomish_Monroe: patching
2024-04-25 03:13:59,280 ERROR [make_v202111_runoff_file] no 2024-04-24 discharge obs for Nisqually_McKenna: patching
2024-04-25 03:13:59,384 ERROR [make_v202111_runoff_file] no 2024-04-24 discharge obs for Greenwater_Greenwater: patching
2024-04-25 03:14:01,092 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-24: /results/forcing/rivers/R202108Dailies_y2024m04d24.nc
2024-04-25 03:14:01,093 INFO [make_v202111_runoff_file] 2024-04-24 runoff file creation completed
2024-04-25 03:14:01,115 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-25 03:14:04,255 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-04-24
2024-04-25 03:14:06,457 INFO [get_onc_ctd] 2024-04-24 ONC SEVIP CTD T&S file created
2024-04-25 03:14:06,467 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-04-25 03:14:06,707 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-04-25 03:14:07,743 INFO [ping_erddap] running in process 2140606
2024-04-25 03:14:07,746 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:14:07,746 INFO [ping_erddap] publishing log messages to tcp://*:5568
2024-04-25 03:14:07,747 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:14:07,795 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-04-25 03:14:07,798 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-25 03:14:21,180 CRITICAL [get_onc_ctd] 2024-04-24 ONC SCVIP CTD T&S file creation failed
2024-04-25 03:14:31,427 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-04-24
2024-04-25 03:15:00,452 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-04-24
2024-04-25 03:15:32,980 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-04-24
2024-04-25 03:15:46,567 INFO [archive_tarball] hindcast *oct20 results files archived to robot.graham
2024-04-25 03:15:46,571 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-25 03:15:49,569 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-04-24
2024-04-25 03:16:08,010 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-04-24
2024-04-25 03:16:17,079 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-04-24
2024-04-25 03:16:26,671 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-04-24; substituting empty dataset
2024-04-25 03:16:26,672 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-04-24
2024-04-25 03:16:35,121 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-24; substituting empty dataset
2024-04-25 03:18:10,495 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240425/06/
2024-04-25 03:18:10,496 INFO [crop_gribs] 2024-04-25 06 GRIBs cropping complete
2024-04-25 03:18:10,498 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-25 03:18:10,940 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-04-25 03:18:13,261 INFO [grib_to_netcdf] running in process 2140638
2024-04-25 03:18:13,286 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:18:13,304 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2024-04-25 03:18:13,316 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:18:13,317 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-25 forecast2 run
2024-04-25 03:18:27,567 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d26.nc
2024-04-25 03:18:31,728 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d27.nc
2024-04-25 03:18:31,736 INFO [grib_to_netcdf] 2024-04-25 NEMO-atmos forcing file for forecast2 created
2024-04-25 03:18:31,738 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-25 03:18:31,990 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-04-25 03:18:31,994 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-04-25 03:18:31,998 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-04-25 03:18:32,007 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-04-25 03:18:33,177 INFO [upload_forcing] running in process 2140678
2024-04-25 03:18:33,178 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:18:33,180 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-04-25 03:18:33,181 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:18:33,195 INFO [upload_forcing] running in process 2140674
2024-04-25 03:18:33,197 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:18:33,198 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-04-25 03:18:33,199 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:18:33,200 INFO [upload_forcing] running in process 2140677
2024-04-25 03:18:33,201 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:18:33,202 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-25 03:18:33,203 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:18:33,210 INFO [upload_forcing] running in process 2140676
2024-04-25 03:18:33,211 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:18:33,212 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-04-25 03:18:33,213 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:18:34,669 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m04d24.nc
2024-04-25 03:18:34,751 INFO [upload_forcing] forecast2 2024-04-25 forcing files upload to optimum-hindcast completed
2024-04-25 03:18:34,755 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-25 03:18:34,980 INFO [upload_forcing] forecast2 2024-04-25 forcing files upload to orcinus-nowcast-agrif completed
2024-04-25 03:18:35,047 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-25 03:18:37,751 INFO [upload_forcing] forecast2 2024-04-25 forcing files upload to arbutus.cloud-nowcast completed
2024-04-25 03:18:37,756 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-25 03:18:37,978 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-25'], host='localhost')
2024-04-25 03:18:39,266 INFO [make_forcing_links] running in process 2140706
2024-04-25 03:18:39,267 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:18:39,268 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-04-25 03:18:39,269 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:18:41,181 INFO [make_forcing_links] forecast2 2024-04-25 forcing file links on arbutus.cloud-nowcast created
2024-04-25 03:18:41,185 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-25 03:18:41,402 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-25'], host='arbutus.cloud-nowcast')
2024-04-25 03:18:41,800 INFO [upload_forcing] forecast2 2024-04-25 forcing files upload to robot.graham completed
2024-04-25 03:18:41,822 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-25 03:18:43,442 INFO [run_NEMO] running in process 13758
2024-04-25 03:18:43,458 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:18:43,459 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-25 03:18:48,245 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-04-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-04-25 03:18:48,264 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-24 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-04-25 03:18:48,649 INFO [get_onc_ferry] 2024-04-24 ONC TWDP ferry data file created
2024-04-25 03:18:48,670 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-04-25 03:18:48,924 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-04-25 03:18:49,784 INFO [ping_erddap] running in process 2140716
2024-04-25 03:18:49,785 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:18:49,787 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-25 03:18:49,788 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:18:49,790 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-04-25 03:18:49,792 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-25 03:18:53,456 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:19:00,429 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/24apr24forecast2_2024-04-25T031859.520293-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-25 03:19:00,449 INFO [run_NEMO] forecast2 NEMO run for 2024-04-25 on arbutus.cloud-nowcast started
2024-04-25 03:19:00,455 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-25 03:19:00,670 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-25 03:19:02,587 INFO [watch_NEMO] running in process 14010
2024-04-25 03:19:02,588 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:19:02,589 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-25 03:19:02,590 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:19:02,659 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-25 03:24:02,660 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17613065 = 2024-04-26 04:42:40 UTC, 15.7% complete
2024-04-25 03:29:02,661 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17613545 = 2024-04-26 10:02:40 UTC, 33.5% complete
2024-04-25 03:34:02,724 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17614033 = 2024-04-26 15:28:00 UTC, 51.6% complete
2024-04-25 03:39:02,673 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17614544 = 2024-04-26 21:08:40 UTC, 70.5% complete
2024-04-25 03:44:02,665 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17615005 = 2024-04-27 02:16:00 UTC, 87.6% complete
2024-04-25 03:49:02,674 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17615340 = 2024-04-27 05:59:20 UTC, 100.0% complete
2024-04-25 03:52:17,001 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-25 03:52:17,415 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-11-01'], host='localhost')
2024-04-25 03:52:17,422 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-25 03:52:17,426 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-25 03:52:18,894 INFO [download_results] running in process 2140818
2024-04-25 03:52:18,896 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:52:18,897 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-25 03:52:18,898 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:52:18,899 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01nov20 to /ocean/sallen/timbucktwo/oxygen
2024-04-25 03:52:22,954 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-25 03:53:21,283 INFO [download_results] hindcast 2020-11-01 results files from optimum-hindcast downloaded
2024-04-25 03:53:21,296 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-25 03:53:21,520 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-11-01'], host='localhost')
2024-04-25 03:53:22,714 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-25 03:54:02,688 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-04-25 03:54:02,717 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-25 03:54:03,106 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-25'], host='arbutus.cloud-nowcast')
2024-04-25 03:54:03,110 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-25'], host='arbutus.cloud-nowcast')
2024-04-25 03:54:03,118 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-24'], host='localhost')
2024-04-25 03:54:04,363 INFO [download_results] running in process 2140870
2024-04-25 03:54:04,367 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:54:04,368 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-25 03:54:04,369 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:54:04,370 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/24apr24 to /results/SalishSea/forecast2.202111
2024-04-25 03:54:10,323 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-25 03:55:58,361 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-25 03:55:58,589 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-25'], host='arbutus.cloud-nowcast')
2024-04-25 03:56:00,611 INFO [run_ww3] running in process 15238
2024-04-25 03:56:00,621 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:56:00,622 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-25 03:56:00,623 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:56:00,627 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2024-04-25T035600.605949-0700
2024-04-25 03:56:00,628 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2024-04-25T035600.605949-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-25 03:56:00,636 INFO [run_ww3] forecast2 WaveWatch3 run for 2024-04-25 on arbutus.cloud-nowcast started
2024-04-25 03:56:00,648 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-25 03:56:00,893 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-25 03:56:02,866 INFO [watch_ww3] running in process 15324
2024-04-25 03:56:02,867 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 03:56:02,868 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-04-25 03:56:02,869 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-25 03:56:02,903 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-25 04:01:02,904 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2024-04-25 04:06:02,916 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete
2024-04-25 04:11:02,917 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 309, 57.2% complete
2024-04-25 04:16:02,937 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 417, 77.2% complete
2024-04-25 04:19:15,552 INFO [download_results] forecast2 2024-04-24 results files from arbutus.cloud-nowcast downloaded
2024-04-25 04:19:15,557 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-25 04:19:15,780 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-04-24'], host='localhost')
2024-04-25 04:19:19,965 INFO [make_CHS_currents_file] running in process 2140972
2024-04-25 04:19:19,967 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:19:19,968 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-04-25 04:19:19,969 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:20:01,908 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-24 for forecast2
2024-04-25 04:20:01,910 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-25 04:20:02,096 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-04-24'], host='localhost')
2024-04-25 04:20:03,229 INFO [update_forecast_datasets] running in process 2140997
2024-04-25 04:20:03,230 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:20:03,231 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-04-25 04:20:03,232 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:20:03,233 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/24apr24 run
2024-04-25 04:21:02,908 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 522, 96.7% complete
2024-04-25 04:24:33,772 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/24apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-25 04:24:33,773 INFO [update_forecast_datasets] nemo 2024-04-24 forecast2 rolling forecast datasets updated
2024-04-25 04:24:33,776 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-25 04:24:34,026 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-25 04:24:34,031 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-04-24'], host='localhost')
2024-04-25 04:24:34,035 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-04-24'], host='localhost')
2024-04-25 04:24:35,280 INFO [ping_erddap] running in process 2141071
2024-04-25 04:24:35,281 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:24:35,282 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-25 04:24:35,283 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:24:35,318 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-25 04:24:35,320 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-25 04:24:37,126 INFO [make_surface_current_tiles] running in process 2141074
2024-04-25 04:24:37,128 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:24:37,129 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2024-04-25 04:24:37,130 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:24:37,184 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-24 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/24apr24
2024-04-25 04:24:42,547 INFO [make_plots] running in process 2141073
2024-04-25 04:24:42,548 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:24:42,550 INFO [make_plots] publishing log messages to tcp://*:5564
2024-04-25 04:24:42,550 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:24:43,610 INFO [make_plots] preparing render list for 2024-04-24 NEMO forecast2 publish figures
2024-04-25 04:24:43,740 INFO [make_plots] added Website_thumbnail figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,742 INFO [make_plots] added Threshold_website figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,743 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,744 INFO [make_plots] added Boundary Bay figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,745 INFO [make_plots] added Campbell River figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,746 INFO [make_plots] added Cherry Point figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,746 INFO [make_plots] added Friday Harbor figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,747 INFO [make_plots] added Halfmoon Bay figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,749 INFO [make_plots] added Nanaimo figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,750 INFO [make_plots] added Neah Bay figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,751 INFO [make_plots] added New Westminster figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,751 INFO [make_plots] added Patricia Bay figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,752 INFO [make_plots] added Point Atkinson figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,753 INFO [make_plots] added Port Renfrew figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,756 INFO [make_plots] added Sand Heads figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,757 INFO [make_plots] added Sandy Cove figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,758 INFO [make_plots] added Squamish figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,759 INFO [make_plots] added Victoria figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,759 INFO [make_plots] added Woodwards Landing figure to 2024-04-24 NEMO forecast2 publish render list
2024-04-25 04:24:43,760 INFO [make_plots] starting to render nemo forecast2 publish 24apr24 figures
2024-04-25 04:26:02,909 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-25 04:26:02,913 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-25 04:26:03,469 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-25'], host='localhost')
2024-04-25 04:26:05,683 INFO [download_wwatch3_results] running in process 2141471
2024-04-25 04:26:05,685 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:26:05,686 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2024-04-25 04:26:05,687 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:28:25,974 INFO [download_wwatch3_results] forecast2 2024-04-25 results files from arbutus.cloud-nowcast downloaded
2024-04-25 04:28:25,982 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-25 04:28:26,419 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-04-25'], host='localhost')
2024-04-25 04:28:28,251 INFO [update_forecast_datasets] running in process 2141722
2024-04-25 04:28:28,252 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:28:28,254 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2024-04-25 04:28:28,255 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:28:28,256 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/25apr24 run
2024-04-25 04:28:28,302 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/25apr24 run
2024-04-25 04:29:10,943 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/25apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-25 04:29:10,957 INFO [update_forecast_datasets] wwatch3 2024-04-25 forecast2 rolling forecast datasets updated
2024-04-25 04:29:10,963 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-25 04:29:11,295 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-25 04:29:12,810 INFO [ping_erddap] running in process 2141917
2024-04-25 04:29:12,833 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:29:12,833 INFO [ping_erddap] publishing log messages to tcp://*:5567
2024-04-25 04:29:12,861 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:29:12,893 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-25 04:29:12,934 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-25 04:29:13,058 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-04-25'], host='localhost')
2024-04-25 04:29:21,301 INFO [make_plots] running in process 2141933
2024-04-25 04:29:21,301 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:29:21,302 INFO [make_plots] publishing log messages to tcp://*:5567
2024-04-25 04:29:21,305 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:29:21,306 INFO [make_plots] preparing render list for 2024-04-25 WaveWatch3 forecast2 publish figures
2024-04-25 04:29:21,307 INFO [make_plots] added Halibut Bank figure to 2024-04-25 WaveWatch3 forecast2 publish render list
2024-04-25 04:29:21,308 INFO [make_plots] added Sentry Shoal figure to 2024-04-25 WaveWatch3 forecast2 publish render list
2024-04-25 04:29:21,309 INFO [make_plots] starting to render wwatch3 forecast2 publish 25apr24 figures
2024-04-25 04:29:21,834 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-25 04:29:35,106 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-25 04:29:44,569 INFO [make_plots] finished rendering wwatch3 forecast2 publish 25apr24 figures
2024-04-25 04:29:44,570 INFO [make_plots] wwatch3 publish plots for 2024-04-25 forecast2 completed
2024-04-25 04:29:44,573 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-25 04:40:16,391 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/24apr24
2024-04-25 04:44:17,313 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-25 04:44:17,314 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-25 04:44:42,783 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/24apr24
2024-04-25 04:44:42,792 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-24 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/24apr24
2024-04-25 04:44:42,794 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-24 forecast2 completed
2024-04-25 04:44:43,105 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-25 04:44:55,987 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:46:48,970 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:46:49,374 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:47:27,180 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:48:42,223 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:49:20,704 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:49:58,553 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:50:36,665 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:51:14,659 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:51:52,888 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:52:30,953 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:52:31,374 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:53:08,120 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:53:45,473 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-20 00:00:00Z to 2024-04-27 05:55:00Z
2024-04-25 04:54:24,232 INFO [make_plots] finished rendering nemo forecast2 publish 24apr24 figures
2024-04-25 04:54:24,235 INFO [make_plots] nemo publish plots for 2024-04-24 forecast2 completed
2024-04-25 04:54:24,244 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-25 04:54:24,621 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-04-24'], host='localhost')
2024-04-25 04:54:28,958 INFO [make_feeds] running in process 2143321
2024-04-25 04:54:28,959 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:54:28,960 INFO [make_feeds] publishing log messages to tcp://*:5562
2024-04-25 04:54:28,961 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:54:59,102 INFO [make_feeds] ATOM feeds for 2024-04-24 forecast2 run completed
2024-04-25 04:54:59,106 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-25 04:54:59,487 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-04-25 04:55:00,384 INFO [clear_checklist] running in process 2143347
2024-04-25 04:55:00,386 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:55:00,387 INFO [clear_checklist] publishing log messages to tcp://*:5562
2024-04-25 04:55:00,388 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:55:00,389 INFO [clear_checklist] requesting that manager clear system state checklist
2024-04-25 04:55:00,392 INFO [manager] writing checklist to log file
2024-04-25 04:55:00,471 INFO [manager] checklist cleared
2024-04-25 04:55:00,474 INFO [clear_checklist] nowcast system checklist cleared
2024-04-25 04:55:00,477 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-04-25 04:55:01,562 INFO [rotate_logs] running in process 2143352
2024-04-25 04:55:01,563 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-25 04:55:01,564 INFO [rotate_logs] publishing log messages to tcp://*:5562
2024-04-25 04:55:01,565 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-04-25 04:55:01,566 INFO [rotate_logs] rotating log files