2024-04-20 04:43:17,558 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-04-20 04:43:17,565 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-04-20 04:43:17,571 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-04-20 04:43:17,572 INFO [rotate_logs] log files rotated
2024-04-20 04:43:17,577 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-04-20 08:30:52,592 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 08:32:37,686 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 09:02:23,495 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240420/12/
2024-04-20 09:02:23,496 INFO [collect_weather] 2024-04-20 2.5km weather forecast 12 collection complete
2024-04-20 09:02:23,499 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-20 09:02:23,695 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-04-19'], host='localhost')
2024-04-20 09:02:23,700 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-04-19'], host='localhost')
2024-04-20 09:02:23,705 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-04-19'], host='localhost')
2024-04-20 09:02:23,709 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-04-19'], host='localhost')
2024-04-20 09:02:23,715 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-04-20 09:02:23,719 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-04-20 09:02:23,725 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-04-20 09:02:23,734 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-04-20 09:02:23,738 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-04-20 09:02:24,970 INFO [collect_weather] running in process 2070470
2024-04-20 09:02:24,970 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:24,971 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-04-20 09:02:24,972 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:24,974 INFO [collect_NeahBay_ssh] running in process 2070468
2024-04-20 09:02:24,975 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:24,976 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-04-20 09:02:24,977 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:24,978 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-04-20
2024-04-20 09:02:24,984 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-04-20 09:02:25,253 INFO [download_live_ocean] running in process 2070469
2024-04-20 09:02:25,254 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:25,255 INFO [download_live_ocean] publishing log messages to tcp://*:5565
2024-04-20 09:02:25,256 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:25,256 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-04-20
2024-04-20 09:02:25,667 INFO [make_turbidity_file] running in process 2070467
2024-04-20 09:02:25,669 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:25,670 INFO [make_turbidity_file] publishing log messages to tcp://*:5566
2024-04-20 09:02:25,671 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:25,672 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-04-20
2024-04-20 09:02:25,689 INFO [collect_river_data] running in process 2070464
2024-04-20 09:02:25,690 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:25,690 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-04-20 09:02:25,691 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:25,692 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-04-19
2024-04-20 09:02:25,716 INFO [collect_river_data] running in process 2070466
2024-04-20 09:02:25,718 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:25,719 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-04-20 09:02:25,719 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:25,720 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-04-19
2024-04-20 09:02:25,751 INFO [collect_river_data] running in process 2070465
2024-04-20 09:02:25,753 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:25,754 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-20 09:02:25,755 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:25,755 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-04-19
2024-04-20 09:02:25,770 INFO [collect_river_data] running in process 2070462
2024-04-20 09:02:25,771 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:25,773 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-04-20 09:02:25,775 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:25,777 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-04-19
2024-04-20 09:02:25,913 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-04-19&endDT=2024-04-19 "HTTP/1.1 200 OK"
2024-04-20 09:02:25,918 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-04-19 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-04-20 09:02:25,919 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-04-19 completed
2024-04-20 09:02:25,921 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-20 09:02:25,933 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-04-19&endDT=2024-04-19 "HTTP/1.1 200 OK"
2024-04-20 09:02:25,938 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-04-19 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-04-20 09:02:25,939 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-04-19 completed
2024-04-20 09:02:25,941 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-20 09:02:26,082 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-04-19&endDT=2024-04-19 "HTTP/1.1 200 OK"
2024-04-20 09:02:26,088 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-04-19 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-04-20 09:02:26,089 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-04-19 completed
2024-04-20 09:02:26,092 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-04-19&endDT=2024-04-19 "HTTP/1.1 200 OK"
2024-04-20 09:02:26,092 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-20 09:02:26,096 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-04-19 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-04-20 09:02:26,096 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-04-19 completed
2024-04-20 09:02:26,099 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-20 09:02:26,102 INFO [crop_gribs] running in process 2070471
2024-04-20 09:02:26,103 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:26,104 INFO [crop_gribs] publishing log messages to tcp://*:5572
2024-04-20 09:02:26,105 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:26,106 INFO [crop_gribs] cropping 2024-04-20 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-04-20 09:02:26,113 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240420/18/
2024-04-20 09:02:26,510 INFO [download_live_ocean] downloaded 10097560 bytes from https://liveocean.apl.uw.edu/output/f2024.04.20/ubc.nc to /results/forcing/LiveOcean/downloaded/20240420/low_passed_UBC.nc
2024-04-20 09:02:26,511 INFO [collect_NeahBay_ssh] 2024-04-20T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-04-20 09:02:26,512 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-04-20 09:02:26,514 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-20 09:02:26,518 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 09:02:27,556 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240420/low_passed_UBC.nc
2024-04-20 09:02:27,557 INFO [download_live_ocean] 2024-04-20 Live Ocean file for Salish Sea western boundary downloaded
2024-04-20 09:02:27,559 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-04-20 09:02:27,755 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-04-20'], host='localhost')
2024-04-20 09:02:28,590 INFO [make_turbidity_file] 2024-04-20 Fraser River turbidity file creation complete
2024-04-20 09:02:28,591 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-04-20 09:02:28,969 INFO [make_ssh_files] running in process 2070622
2024-04-20 09:02:28,970 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:28,971 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2024-04-20 09:02:28,972 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:28,973 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-04-20 NOAA Neah Bay 06Z observation and forecast values
2024-04-20 09:02:29,563 INFO [make_live_ocean_files] running in process 2070642
2024-04-20 09:02:29,564 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:29,566 INFO [make_live_ocean_files] publishing log messages to tcp://*:5565
2024-04-20 09:02:29,567 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:29,568 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-04-20 Live Ocean run
2024-04-20 09:02:33,856 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d16.nc
2024-04-20 09:02:34,031 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc
2024-04-20 09:02:34,204 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d18.nc
2024-04-20 09:02:34,380 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d19.nc
2024-04-20 09:02:34,572 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d20.nc
2024-04-20 09:02:34,750 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d21.nc
2024-04-20 09:02:34,930 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d22.nc
2024-04-20 09:02:35,118 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d23.nc
2024-04-20 09:02:35,234 INFO [make_ssh_files] sea surface height boundary file for 2024-04-20 nowcast run created
2024-04-20 09:02:35,237 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-20 09:02:35,241 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-20 09:02:35,245 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-20 09:02:36,272 INFO [make_runoff_file] running in process 2070680
2024-04-20 09:02:36,273 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:36,274 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-04-20 09:02:36,275 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:36,299 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-20 09:02:36,397 INFO [make_runoff_file] 2024-04-20 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-20 09:02:36,400 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-20 09:02:36,888 INFO [make_v202111_runoff_file] running in process 2070678
2024-04-20 09:02:36,890 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:02:36,891 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564
2024-04-20 09:02:36,892 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:02:36,893 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-19
2024-04-20 09:02:39,221 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-19: /results/forcing/rivers/R202108Dailies_y2024m04d19.nc
2024-04-20 09:02:39,222 INFO [make_v202111_runoff_file] 2024-04-19 runoff file creation completed
2024-04-20 09:02:39,224 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-20 09:03:18,553 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2024-04-20 09:03:18,554 INFO [make_live_ocean_files] 2024-04-20 Live Ocean western boundary conditions files created
2024-04-20 09:03:18,556 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-04-20 09:11:23,094 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240420/12/
2024-04-20 09:11:23,096 INFO [crop_gribs] 2024-04-20 12 GRIBs cropping complete
2024-04-20 09:11:23,098 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-20 09:11:23,297 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-04-20 09:11:24,999 INFO [grib_to_netcdf] running in process 2070752
2024-04-20 09:11:25,000 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:11:25,001 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2024-04-20 09:11:25,003 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:11:25,004 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-20 nowcast and forecast runs
2024-04-20 09:11:37,488 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m04d20.nc
2024-04-20 09:11:48,059 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d21.nc
2024-04-20 09:11:54,054 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d22.nc
2024-04-20 09:11:54,070 INFO [grib_to_netcdf] 2024-04-20 NEMO-atmos forcing file for nowcast+ created
2024-04-20 09:11:54,073 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-20 09:11:54,081 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-04-20 09:11:54,086 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-04-20 09:11:54,089 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-04-20 09:11:54,092 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-04-20 09:11:54,096 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-04-20 09:11:54,976 INFO [ping_erddap] running in process 2070794
2024-04-20 09:11:54,978 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:11:54,979 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-04-20 09:11:54,980 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:11:54,982 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-04-20 09:11:54,984 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-20 09:11:55,030 INFO [upload_forcing] running in process 2070789
2024-04-20 09:11:55,032 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:11:55,033 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-04-20 09:11:55,034 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:11:55,045 INFO [upload_forcing] running in process 2070793
2024-04-20 09:11:55,046 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:11:55,047 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-20 09:11:55,048 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:11:55,049 INFO [upload_forcing] running in process 2070791
2024-04-20 09:11:55,050 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:11:55,051 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-20 09:11:55,052 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:11:55,102 INFO [upload_forcing] running in process 2070792
2024-04-20 09:11:55,103 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:11:55,104 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-20 09:11:55,105 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:11:57,639 INFO [upload_forcing] nowcast+ 2024-04-20 forcing files upload to optimum-hindcast completed
2024-04-20 09:11:57,643 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-20 09:11:57,765 INFO [upload_forcing] nowcast+ 2024-04-20 forcing files upload to orcinus-nowcast-agrif completed
2024-04-20 09:11:57,771 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-20 09:12:04,467 INFO [upload_forcing] nowcast+ 2024-04-20 forcing files upload to robot.graham completed
2024-04-20 09:12:04,473 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-20 09:12:08,509 INFO [upload_forcing] nowcast+ 2024-04-20 forcing files upload to arbutus.cloud-nowcast completed
2024-04-20 09:12:08,515 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-20 09:12:08,526 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 09:12:09,551 INFO [make_forcing_links] running in process 2070828
2024-04-20 09:12:09,554 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:12:09,556 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-04-20 09:12:09,557 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:12:10,730 INFO [make_forcing_links] nowcast+ 2024-04-20 forcing file links on arbutus.cloud-nowcast created
2024-04-20 09:12:10,734 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-20 09:12:10,745 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-20'], host='arbutus.cloud-nowcast')
2024-04-20 09:12:12,887 INFO [run_NEMO] running in process 2831
2024-04-20 09:12:12,888 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:12:12,890 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-20 09:12:12,954 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:12:13,703 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/20apr24nowcast_2024-04-20T091212.931869-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-20 09:12:13,722 INFO [run_NEMO] nowcast NEMO run for 2024-04-20 on arbutus.cloud-nowcast started
2024-04-20 09:12:13,727 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-20 09:12:13,737 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-20 09:12:15,683 INFO [watch_NEMO] running in process 3083
2024-04-20 09:12:15,684 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:12:15,685 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-20 09:12:15,687 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:12:15,753 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-20 09:17:15,756 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17600129 = 2024-04-20 04:58:40 UTC, 20.8% complete
2024-04-20 09:22:15,756 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17600645 = 2024-04-20 10:42:40 UTC, 44.7% complete
2024-04-20 09:27:15,754 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17601134 = 2024-04-20 16:08:40 UTC, 67.3% complete
2024-04-20 09:32:15,753 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17601619 = 2024-04-20 21:32:00 UTC, 89.8% complete
2024-04-20 09:37:15,756 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-04-20 09:37:15,760 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-20 09:37:15,968 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 09:37:15,975 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 09:37:16,985 INFO [download_results] running in process 2071077
2024-04-20 09:37:16,986 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:37:16,988 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-20 09:37:16,989 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:37:16,990 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/20apr24 to /results/SalishSea/nowcast-blue.202111
2024-04-20 09:37:16,992 INFO [make_forcing_links] running in process 2071076
2024-04-20 09:37:16,993 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:37:16,994 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-04-20 09:37:16,995 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:37:17,939 INFO [make_forcing_links] ssh 2024-04-20 forcing file links on arbutus.cloud-nowcast created
2024-04-20 09:37:17,943 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-20 09:37:17,954 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-20'], host='arbutus.cloud-nowcast')
2024-04-20 09:37:20,072 INFO [run_NEMO] running in process 3370
2024-04-20 09:37:20,073 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:37:20,074 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-20 09:37:20,075 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:37:21,029 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/20apr24forecast_2024-04-20T093720.146494-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-20 09:37:21,050 INFO [run_NEMO] forecast NEMO run for 2024-04-20 on arbutus.cloud-nowcast started
2024-04-20 09:37:21,053 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-20 09:37:21,063 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-20 09:37:23,211 INFO [watch_NEMO] running in process 3622
2024-04-20 09:37:23,212 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:37:23,214 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-20 09:37:23,215 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:37:23,305 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-20 09:42:23,311 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17602278 = 2024-04-21 04:51:20 UTC, 13.5% complete
2024-04-20 09:47:23,303 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17602767 = 2024-04-21 10:17:20 UTC, 28.6% complete
2024-04-20 09:52:23,317 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17603236 = 2024-04-21 15:30:00 UTC, 43.1% complete
2024-04-20 09:57:23,306 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17603714 = 2024-04-21 20:48:40 UTC, 57.8% complete
2024-04-20 09:57:23,877 INFO [download_results] nowcast 2024-04-20 results files from arbutus.cloud-nowcast downloaded
2024-04-20 09:57:23,882 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 09:57:23,890 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 09:57:23,895 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-04-19'], host='localhost')
2024-04-20 09:57:23,901 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 09:57:25,684 INFO [make_CHS_currents_file] running in process 2071383
2024-04-20 09:57:25,685 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:57:25,686 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-04-20 09:57:25,687 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:57:26,966 INFO [make_plots] running in process 2071380
2024-04-20 09:57:26,967 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:57:26,968 INFO [make_plots] publishing log messages to tcp://*:5565
2024-04-20 09:57:26,970 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:57:26,996 INFO [make_plots] running in process 2071382
2024-04-20 09:57:27,012 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:57:27,018 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-20 09:57:27,019 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:57:27,594 INFO [make_plots] preparing render list for 2024-04-19 NEMO nowcast-blue comparison figures
2024-04-20 09:57:27,598 INFO [make_plots] preparing render list for 2024-04-20 NEMO nowcast-blue research figures
2024-04-20 09:57:28,673 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-blue research render list
2024-04-20 09:57:29,330 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-blue research render list
2024-04-20 09:57:29,331 INFO [make_plots] added Currents_sections_and_surface figure to 2024-04-20 NEMO nowcast-blue research render list
2024-04-20 09:57:29,333 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-04-20 NEMO nowcast-blue research render list
2024-04-20 09:57:29,334 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-04-20 NEMO nowcast-blue research render list
2024-04-20 09:57:29,335 INFO [make_plots] starting to render nemo nowcast research 20apr24 figures
2024-04-20 09:57:46,691 INFO [make_plots] added SH_wind figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,692 INFO [make_plots] added Compare_VENUS_East figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,693 INFO [make_plots] added Compare_VENUS_Central figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,694 INFO [make_plots] added Central_ADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,695 INFO [make_plots] added Central_depavADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,696 INFO [make_plots] added Central_timeavADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,698 INFO [make_plots] added East_ADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,698 INFO [make_plots] added East_depavADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,699 INFO [make_plots] added East_timeavADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,700 INFO [make_plots] added ddl_ADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,701 INFO [make_plots] added ddl_depavADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,701 INFO [make_plots] added ddl_timeavADCP figure to 2024-04-19 NEMO nowcast-blue comparison render list
2024-04-20 09:57:46,702 INFO [make_plots] starting to render nemo nowcast comparison 19apr24 figures
2024-04-20 09:57:56,643 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-20 for nowcast
2024-04-20 09:57:56,645 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-20 09:58:13,577 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 09:58:13,786 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-01'], host='localhost')
2024-04-20 09:58:13,791 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 09:58:13,797 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 09:58:14,833 INFO [download_results] running in process 2071476
2024-04-20 09:58:14,834 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 09:58:14,835 INFO [download_results] publishing log messages to tcp://*:5567
2024-04-20 09:58:14,836 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 09:58:14,837 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01oct19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 09:58:18,979 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 09:58:49,250 INFO [make_plots] finished rendering nemo nowcast research 20apr24 figures
2024-04-20 09:58:49,252 INFO [make_plots] nemo research plots for 2024-04-20 nowcast completed
2024-04-20 09:58:49,270 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-20 09:59:03,315 INFO [download_results] hindcast 2019-10-01 results files from optimum-hindcast downloaded
2024-04-20 09:59:03,330 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 09:59:03,345 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-01'], host='localhost')
2024-04-20 09:59:04,333 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 09:59:47,028 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-20 09:59:47,502 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-20 09:59:47,988 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-20 09:59:48,463 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-20 09:59:48,938 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-20 09:59:49,413 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-20 09:59:49,886 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-20 09:59:50,358 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-20 09:59:50,829 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-20 09:59:50,831 INFO [make_plots] finished rendering nemo nowcast comparison 19apr24 figures
2024-04-20 09:59:50,936 INFO [make_plots] nemo comparison plots for 2024-04-19 nowcast completed
2024-04-20 09:59:50,940 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-20 10:02:23,299 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17604158 = 2024-04-22 01:44:40 UTC, 71.5% complete
2024-04-20 10:07:23,298 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17604616 = 2024-04-22 06:50:00 UTC, 85.7% complete
2024-04-20 10:12:23,297 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17605080 = 2024-04-22 11:59:20 UTC, 100.0% complete
2024-04-20 10:17:23,300 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-04-20 10:17:23,304 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-20 10:17:23,670 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-20'], host='arbutus.cloud-nowcast')
2024-04-20 10:17:23,677 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-20'], host='arbutus.cloud-nowcast')
2024-04-20 10:17:23,682 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:17:23,688 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:17:23,692 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:17:23,698 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:17:23,701 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:17:24,717 INFO [upload_forcing] running in process 2071631
2024-04-20 10:17:24,718 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:24,719 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-04-20 10:17:24,720 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:24,722 INFO [download_results] running in process 2071634
2024-04-20 10:17:24,723 INFO [upload_forcing] running in process 2071629
2024-04-20 10:17:24,724 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:24,725 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:24,726 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-20 10:17:24,727 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-20 10:17:24,728 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:24,729 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:24,730 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/20apr24 to /results/SalishSea/forecast.202111
2024-04-20 10:17:24,733 INFO [upload_forcing] running in process 2071633
2024-04-20 10:17:24,734 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:24,735 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-20 10:17:24,736 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:24,738 INFO [upload_forcing] running in process 2071632
2024-04-20 10:17:24,739 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:24,740 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-20 10:17:24,740 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:25,443 INFO [upload_forcing] turbidity 2024-04-20 forcing files upload to optimum-hindcast completed
2024-04-20 10:17:25,447 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-20 10:17:25,650 INFO [upload_forcing] turbidity 2024-04-20 forcing files upload to orcinus-nowcast-agrif completed
2024-04-20 10:17:25,655 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-20 10:17:25,674 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:17:25,912 INFO [upload_forcing] turbidity 2024-04-20 forcing files upload to arbutus.cloud-nowcast completed
2024-04-20 10:17:25,916 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-20 10:17:25,934 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:17:26,673 INFO [make_forcing_links] running in process 2071665
2024-04-20 10:17:26,675 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:26,675 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-04-20 10:17:26,676 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:26,938 INFO [make_forcing_links] running in process 2071667
2024-04-20 10:17:26,939 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:26,940 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2024-04-20 10:17:26,941 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:27,663 INFO [make_forcing_links] nowcast-agrif 2024-04-20 forcing file links on orcinus-nowcast-agrif created
2024-04-20 10:17:27,669 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-20 10:17:27,690 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:17:28,688 INFO [run_NEMO_agrif] running in process 2071681
2024-04-20 10:17:28,689 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:28,690 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2024-04-20 10:17:28,692 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:28,785 INFO [upload_forcing] turbidity 2024-04-20 forcing files upload to robot.graham completed
2024-04-20 10:17:28,789 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-20 10:17:30,751 INFO [make_forcing_links] nowcast-green 2024-04-20 forcing file links on arbutus.cloud-nowcast created
2024-04-20 10:17:30,756 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-20 10:17:30,776 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-20'], host='arbutus.cloud-nowcast')
2024-04-20 10:17:32,027 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-20 10:17:34,562 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/20apr24nowcast-green_2024-04-20T101733.502669-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-20 10:17:34,582 INFO [run_NEMO] nowcast-green NEMO run for 2024-04-20 on arbutus.cloud-nowcast started
2024-04-20 10:17:34,588 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-20 10:17:34,610 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-04-20 10:17:37,747 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-20 10:17:54,020 INFO [run_NEMO_agrif] job id for 20apr24nowcast-agrif: 11092005.orca2.ibb
2024-04-20 10:17:54,022 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-04-20 queued on orcinus-nowcast-agrif
2024-04-20 10:17:54,026 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-04-20 10:17:54,046 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11092005.orca2.ibb'], host='localhost')
2024-04-20 10:17:55,045 INFO [watch_NEMO_agrif] running in process 2071700
2024-04-20 10:17:55,046 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:17:55,047 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2024-04-20 10:17:55,048 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:17:55,890 INFO [watch_NEMO_agrif] watching 20apr24nowcast-agrif job 11092005 on orcinus-nowcast-agrif
2024-04-20 10:17:56,985 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-04-20 10:18:11,388 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-20 10:18:11,410 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-20'], host='arbutus.cloud-nowcast')
2024-04-20 10:18:17,854 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-20 10:18:17,877 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-20 10:18:25,051 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-20 10:22:38,247 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17599846 = 2024-04-20 01:50:00 UTC, 7.6% complete
2024-04-20 10:22:57,440 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7620641 = 2024-04-20 01:46:40 UTC, 7.4% complete
2024-04-20 10:23:25,341 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 99, 22.9% complete
2024-04-20 10:27:38,334 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17600038 = 2024-04-20 03:58:00 UTC, 16.5% complete
2024-04-20 10:27:57,926 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7620838 = 2024-04-20 03:58:00 UTC, 16.5% complete
2024-04-20 10:28:25,317 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 201, 46.5% complete
2024-04-20 10:32:38,373 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17600227 = 2024-04-20 06:04:00 UTC, 25.3% complete
2024-04-20 10:32:58,408 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7621034 = 2024-04-20 06:08:40 UTC, 25.6% complete
2024-04-20 10:33:25,346 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 306, 70.8% complete
2024-04-20 10:37:38,337 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17600412 = 2024-04-20 08:07:20 UTC, 33.9% complete
2024-04-20 10:37:58,855 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7621233 = 2024-04-20 08:21:20 UTC, 34.8% complete
2024-04-20 10:38:25,339 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 408, 94.4% complete
2024-04-20 10:42:38,284 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17600600 = 2024-04-20 10:12:40 UTC, 42.6% complete
2024-04-20 10:42:59,303 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7621431 = 2024-04-20 10:33:20 UTC, 44.0% complete
2024-04-20 10:43:25,359 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-20 10:43:25,362 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-20 10:43:25,575 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:43:25,581 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-20'], host='arbutus.cloud-nowcast')
2024-04-20 10:43:26,499 INFO [download_wwatch3_results] running in process 2071739
2024-04-20 10:43:26,500 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:43:26,502 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-04-20 10:43:26,503 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:43:39,144 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-20 10:43:39,166 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-20 10:43:51,387 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-20 10:47:38,639 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17600793 = 2024-04-20 12:21:20 UTC, 51.5% complete
2024-04-20 10:47:59,759 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7621630 = 2024-04-20 12:46:00 UTC, 53.2% complete
2024-04-20 10:48:52,001 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2024-04-20 10:48:56,803 INFO [download_results] forecast 2024-04-20 results files from arbutus.cloud-nowcast downloaded
2024-04-20 10:48:56,808 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 10:48:56,830 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:48:57,871 INFO [download_wwatch3_results] nowcast 2024-04-20 results files from arbutus.cloud-nowcast downloaded
2024-04-20 10:48:57,874 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-20 10:48:59,080 INFO [make_CHS_currents_file] running in process 2071760
2024-04-20 10:48:59,081 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:48:59,083 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-04-20 10:48:59,084 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:49:39,521 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-20 for forecast
2024-04-20 10:49:39,525 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-20 10:49:39,547 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:49:40,392 INFO [update_forecast_datasets] running in process 2071786
2024-04-20 10:49:40,393 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:49:40,394 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-04-20 10:49:40,396 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:49:40,397 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/20apr24 run
2024-04-20 10:49:40,861 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/20apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-20 10:49:40,862 INFO [update_forecast_datasets] nemo 2024-04-20 forecast rolling forecast datasets updated
2024-04-20 10:49:40,865 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-20 10:49:40,879 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-20 10:49:40,884 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:49:40,888 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 10:49:41,754 INFO [ping_erddap] running in process 2071791
2024-04-20 10:49:41,757 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:49:41,758 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-04-20 10:49:41,758 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:49:41,767 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-20 10:49:41,769 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-20 10:49:42,896 INFO [make_surface_current_tiles] running in process 2071794
2024-04-20 10:49:42,897 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:49:42,898 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-04-20 10:49:42,899 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:49:42,901 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-20 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/20apr24
2024-04-20 10:49:44,226 INFO [make_plots] running in process 2071793
2024-04-20 10:49:44,231 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 10:49:44,237 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-20 10:49:44,238 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-20 10:49:44,953 INFO [make_plots] preparing render list for 2024-04-20 NEMO forecast publish figures
2024-04-20 10:49:45,085 INFO [make_plots] added Website_thumbnail figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,087 INFO [make_plots] added Threshold_website figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,087 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,088 INFO [make_plots] added Boundary Bay figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,089 INFO [make_plots] added Campbell River figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,090 INFO [make_plots] added Cherry Point figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,091 INFO [make_plots] added Friday Harbor figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,092 INFO [make_plots] added Halfmoon Bay figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,092 INFO [make_plots] added Nanaimo figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,093 INFO [make_plots] added Neah Bay figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,094 INFO [make_plots] added New Westminster figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,094 INFO [make_plots] added Patricia Bay figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,095 INFO [make_plots] added Point Atkinson figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,096 INFO [make_plots] added Port Renfrew figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,096 INFO [make_plots] added Sand Heads figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,097 INFO [make_plots] added Sandy Cove figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,098 INFO [make_plots] added Squamish figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,099 INFO [make_plots] added Victoria figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,102 INFO [make_plots] added Woodwards Landing figure to 2024-04-20 NEMO forecast publish render list
2024-04-20 10:49:45,103 INFO [make_plots] starting to render nemo forecast publish 20apr24 figures
2024-04-20 10:52:37,669 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17600987 = 2024-04-20 14:30:40 UTC, 60.5% complete
2024-04-20 10:53:00,246 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7621828 = 2024-04-20 14:58:00 UTC, 62.4% complete
2024-04-20 10:53:50,702 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 198, 30.6% complete
2024-04-20 10:57:37,679 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17601173 = 2024-04-20 16:34:40 UTC, 69.1% complete
2024-04-20 10:58:00,740 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7622025 = 2024-04-20 17:09:20 UTC, 71.5% complete
2024-04-20 10:58:50,715 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 300, 46.3% complete
2024-04-20 11:02:37,692 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17601361 = 2024-04-20 18:40:00 UTC, 77.8% complete
2024-04-20 11:03:01,348 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7622223 = 2024-04-20 19:21:20 UTC, 80.7% complete
2024-04-20 11:03:26,486 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/20apr24
2024-04-20 11:03:50,726 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 405, 62.5% complete
2024-04-20 11:07:19,263 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/20apr24
2024-04-20 11:07:19,273 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-20 forecast into /results/nowcast-sys/figures/surface_currents/forecast/20apr24
2024-04-20 11:07:19,275 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-20 forecast completed
2024-04-20 11:07:19,620 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-20 11:07:24,382 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-20 11:07:24,384 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-20 11:07:37,698 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17601544 = 2024-04-20 20:42:00 UTC, 86.3% complete
2024-04-20 11:07:59,742 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:08:01,965 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7622422 = 2024-04-20 21:34:00 UTC, 89.9% complete
2024-04-20 11:08:50,732 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 507, 78.2% complete
2024-04-20 11:09:47,506 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:09:47,919 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:10:22,823 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:11:34,152 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:12:09,871 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:12:37,703 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17601725 = 2024-04-20 22:42:40 UTC, 94.7% complete
2024-04-20 11:12:46,424 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:13:02,722 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7622619 = 2024-04-20 23:45:20 UTC, 99.0% complete
2024-04-20 11:13:22,129 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:13:50,735 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 606, 93.5% complete
2024-04-20 11:13:58,713 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:14:34,067 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:15:11,288 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:15:11,701 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:15:47,649 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:16:23,017 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-15 00:00:00Z to 2024-04-22 11:55:00Z
2024-04-20 11:16:58,486 INFO [make_plots] finished rendering nemo forecast publish 20apr24 figures
2024-04-20 11:16:58,488 INFO [make_plots] nemo publish plots for 2024-04-20 forecast completed
2024-04-20 11:16:58,497 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-20 11:16:58,678 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:17:01,226 INFO [make_feeds] running in process 2073988
2024-04-20 11:17:01,227 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:17:01,229 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-04-20 11:17:01,230 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:17:28,665 INFO [make_feeds] ATOM feeds for 2024-04-20 forecast run completed
2024-04-20 11:17:28,669 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-20 11:17:37,702 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17601840 = 2024-04-20 23:59:20 UTC, 100.0% complete
2024-04-20 11:18:03,251 INFO [watch_NEMO_agrif] 20apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7622640 = 2024-04-20 23:59:20 UTC, 100.0% complete
2024-04-20 11:18:50,736 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-20 11:18:50,739 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-20 11:18:51,113 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:18:52,113 INFO [download_wwatch3_results] running in process 2074016
2024-04-20 11:18:52,114 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:18:52,115 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-04-20 11:18:52,116 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:21:29,088 INFO [download_wwatch3_results] forecast 2024-04-20 results files from arbutus.cloud-nowcast downloaded
2024-04-20 11:21:29,092 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-20 11:21:29,275 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:21:30,207 INFO [update_forecast_datasets] running in process 2074029
2024-04-20 11:21:30,209 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:21:30,210 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-04-20 11:21:30,211 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:21:30,212 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/20apr24 run
2024-04-20 11:21:30,214 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/20apr24 run
2024-04-20 11:21:30,263 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/20apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-20 11:21:30,264 INFO [update_forecast_datasets] wwatch3 2024-04-20 forecast rolling forecast datasets updated
2024-04-20 11:21:30,266 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-20 11:21:30,448 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-20 11:21:31,292 INFO [ping_erddap] running in process 2074034
2024-04-20 11:21:31,293 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:21:31,295 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-04-20 11:21:31,296 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:21:31,297 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-20 11:21:31,299 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-20 11:21:31,480 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:21:34,401 INFO [make_plots] running in process 2074039
2024-04-20 11:21:34,404 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:21:34,405 INFO [make_plots] publishing log messages to tcp://*:5565
2024-04-20 11:21:34,406 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:21:34,407 INFO [make_plots] preparing render list for 2024-04-20 WaveWatch3 forecast publish figures
2024-04-20 11:21:34,408 INFO [make_plots] added Halibut Bank figure to 2024-04-20 WaveWatch3 forecast publish render list
2024-04-20 11:21:34,409 INFO [make_plots] added Sentry Shoal figure to 2024-04-20 WaveWatch3 forecast publish render list
2024-04-20 11:21:34,409 INFO [make_plots] starting to render wwatch3 forecast publish 20apr24 figures
2024-04-20 11:21:34,619 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-20 11:21:42,879 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-20 11:21:49,306 INFO [make_plots] finished rendering wwatch3 forecast publish 20apr24 figures
2024-04-20 11:21:49,307 INFO [make_plots] wwatch3 publish plots for 2024-04-20 forecast completed
2024-04-20 11:21:49,311 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-20 11:22:37,710 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-04-20 11:22:37,714 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-20 11:22:38,083 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:22:39,128 INFO [download_results] running in process 2074075
2024-04-20 11:22:39,130 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:22:39,131 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-20 11:22:39,132 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:22:39,133 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/20apr24 to /results2/SalishSea/nowcast-green.202111
2024-04-20 11:23:03,563 INFO [watch_NEMO_agrif] NEMO AGRIF run 11092005.orca2.ibb on orcinus-nowcast-agrif completed
2024-04-20 11:23:03,568 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-04-20 11:23:03,942 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:23:04,903 INFO [download_results] running in process 2074082
2024-04-20 11:23:04,904 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:23:04,905 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-20 11:23:04,906 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:23:04,907 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/20apr24 to /results/SalishSea/nowcast-agrif.201702
2024-04-20 11:23:23,306 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 11:23:23,697 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-06'], host='localhost')
2024-04-20 11:23:23,703 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 11:23:23,708 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 11:23:24,868 INFO [download_results] running in process 2074089
2024-04-20 11:23:24,869 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:23:24,870 INFO [download_results] publishing log messages to tcp://*:5567
2024-04-20 11:23:24,872 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:23:24,873 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06oct19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 11:23:29,163 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 11:23:37,195 INFO [download_results] nowcast-agrif 2024-04-20 results files from orcinus-nowcast-agrif downloaded
2024-04-20 11:23:37,204 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 11:23:37,402 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:23:42,807 INFO [make_plots] running in process 2074126
2024-04-20 11:23:42,808 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:23:42,809 INFO [make_plots] publishing log messages to tcp://*:5564
2024-04-20 11:23:42,811 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:23:43,535 INFO [make_plots] preparing render list for 2024-04-20 NEMO nowcast-agrif research figures
2024-04-20 11:23:43,536 INFO [make_plots] added baynes_sound_surface figure to 2024-04-20 NEMO nowcast-agrif research render list
2024-04-20 11:23:43,537 INFO [make_plots] starting to render nemo nowcast-agrif research 20apr24 figures
2024-04-20 11:23:55,546 INFO [make_plots] finished rendering nemo nowcast-agrif research 20apr24 figures
2024-04-20 11:23:55,548 INFO [make_plots] nemo research plots for 2024-04-20 nowcast-agrif completed
2024-04-20 11:23:55,553 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-20 11:24:19,822 INFO [download_results] hindcast 2019-10-06 results files from optimum-hindcast downloaded
2024-04-20 11:24:19,837 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 11:24:20,013 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-06'], host='localhost')
2024-04-20 11:24:21,010 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 11:56:39,066 INFO [download_results] nowcast-green 2024-04-20 results files from arbutus.cloud-nowcast downloaded
2024-04-20 11:56:39,072 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 11:56:39,257 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:56:39,262 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-04-20 11:56:39,266 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:56:39,272 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:56:39,275 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-04-20'], host='localhost')
2024-04-20 11:56:40,586 INFO [ping_erddap] running in process 2074274
2024-04-20 11:56:40,588 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:56:40,590 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-04-20 11:56:40,592 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:56:40,606 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-04-20 11:56:40,610 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-20 11:56:42,693 INFO [make_averaged_dataset] running in process 2074276
2024-04-20 11:56:42,694 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:56:42,695 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2024-04-20 11:56:42,696 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:56:42,696 INFO [make_averaged_dataset] creating day-averaged dataset for 20-Apr-2024 chemistry
2024-04-20 11:56:42,817 INFO [make_averaged_dataset] running in process 2074277
2024-04-20 11:56:42,819 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:56:42,820 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2024-04-20 11:56:42,821 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:56:42,822 INFO [make_averaged_dataset] creating day-averaged dataset for 20-Apr-2024 biology
2024-04-20 11:56:42,822 INFO [make_averaged_dataset] running in process 2074275
2024-04-20 11:56:42,823 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:56:42,825 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2024-04-20 11:56:42,826 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:56:42,827 INFO [make_averaged_dataset] creating day-averaged dataset for 20-Apr-2024 physics
2024-04-20 11:56:45,102 INFO [make_plots] running in process 2074272
2024-04-20 11:56:45,102 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 11:56:45,107 INFO [make_plots] publishing log messages to tcp://*:5567
2024-04-20 11:56:45,111 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-20 11:56:45,864 INFO [make_plots] preparing render list for 2024-04-20 NEMO nowcast-green research figures
2024-04-20 11:56:47,883 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:48,658 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:49,421 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:50,230 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:51,034 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:51,757 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:52,536 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:53,505 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:54,393 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:55,286 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:56,165 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:56,987 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:57,365 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:57,366 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:57,367 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:57,367 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-04-20 NEMO nowcast-green research render list
2024-04-20 11:56:57,368 INFO [make_plots] starting to render nemo nowcast-green research 20apr24 figures
2024-04-20 11:57:42,240 INFO [make_averaged_dataset] day-averaged dataset for 20-Apr-2024 physics created
2024-04-20 11:57:42,243 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-20 11:58:31,958 INFO [make_averaged_dataset] day-averaged dataset for 20-Apr-2024 chemistry created
2024-04-20 11:58:31,962 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-20 11:58:56,212 INFO [make_averaged_dataset] day-averaged dataset for 20-Apr-2024 biology created
2024-04-20 11:58:56,215 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-20 12:53:33,473 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 12:53:33,897 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-11'], host='localhost')
2024-04-20 12:53:33,905 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 12:53:33,909 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 12:53:35,819 INFO [download_results] running in process 2074804
2024-04-20 12:53:35,820 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 12:53:35,821 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-20 12:53:35,822 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 12:53:35,823 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11oct19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 12:53:39,921 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 12:55:25,631 INFO [download_results] hindcast 2019-10-11 results files from optimum-hindcast downloaded
2024-04-20 12:55:25,647 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 12:55:25,828 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-11'], host='localhost')
2024-04-20 12:55:27,059 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 13:23:23,645 INFO [make_plots] finished rendering nemo nowcast-green research 20apr24 figures
2024-04-20 13:23:23,819 INFO [make_plots] nemo research plots for 2024-04-20 nowcast-green completed
2024-04-20 13:23:23,885 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-20 14:18:44,577 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 14:18:44,995 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-16'], host='localhost')
2024-04-20 14:18:45,003 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 14:18:45,009 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 14:18:46,906 INFO [download_results] running in process 2075078
2024-04-20 14:18:46,907 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 14:18:46,909 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-20 14:18:46,910 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 14:18:46,911 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16oct19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 14:18:50,961 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 14:19:54,004 INFO [download_results] hindcast 2019-10-16 results files from optimum-hindcast downloaded
2024-04-20 14:19:54,019 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 14:19:54,236 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-16'], host='localhost')
2024-04-20 14:19:55,572 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 15:09:18,611 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240420/18/
2024-04-20 15:09:18,612 INFO [collect_weather] 2024-04-20 2.5km weather forecast 18 collection complete
2024-04-20 15:09:18,615 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-20 15:09:19,026 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-04-20 15:09:19,033 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-04-20 15:09:19,038 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-04-20 15:09:19,044 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-04-21'], host='localhost')
2024-04-20 15:09:20,380 INFO [download_weather] running in process 2075268
2024-04-20 15:09:20,381 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 15:09:20,382 INFO [download_weather] publishing log messages to tcp://*:5562
2024-04-20 15:09:20,383 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-20 15:09:20,384 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240420
2024-04-20 15:09:20,386 INFO [collect_weather] running in process 2075271
2024-04-20 15:09:20,386 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 15:09:20,388 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-04-20 15:09:20,389 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-20 15:09:20,517 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-04-20 15:09:20,518 INFO [download_weather] running in process 2075270
2024-04-20 15:09:20,519 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 15:09:20,521 INFO [download_weather] publishing log messages to tcp://*:5564
2024-04-20 15:09:20,525 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-20 15:09:20,528 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240420
2024-04-20 15:09:24,035 INFO [crop_gribs] running in process 2075272
2024-04-20 15:09:24,036 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 15:09:24,037 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-04-20 15:09:24,038 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-20 15:09:24,039 INFO [crop_gribs] cropping 2024-04-21 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-04-20 15:09:24,048 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240421/00/
2024-04-20 15:16:24,536 INFO [download_weather] 2024-04-20 1km weather forecast 00 downloads complete
2024-04-20 15:16:24,540 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-20 15:17:06,505 INFO [download_weather] 2024-04-20 1km weather forecast 12 downloads complete
2024-04-20 15:17:06,508 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-20 15:43:55,576 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 15:43:55,980 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-21'], host='localhost')
2024-04-20 15:43:55,987 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 15:43:55,991 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 15:43:57,824 INFO [download_results] running in process 2075496
2024-04-20 15:43:57,825 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 15:43:57,826 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-20 15:43:57,827 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 15:43:57,828 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21oct19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 15:44:02,318 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 15:44:59,826 INFO [download_results] hindcast 2019-10-21 results files from optimum-hindcast downloaded
2024-04-20 15:44:59,851 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 15:45:00,042 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-21'], host='localhost')
2024-04-20 15:45:01,187 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 17:02:27,576 INFO [crop_gribs] crop_gribs 18 has watched for 8h; retrying remaining 1 file(s)
2024-04-20 17:02:30,407 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240420/18/
2024-04-20 17:02:30,407 INFO [crop_gribs] 2024-04-20 18 GRIBs cropping complete
2024-04-20 17:02:30,410 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-20 17:29:07,988 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 17:29:08,400 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-10-26'], host='localhost')
2024-04-20 17:29:08,406 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 17:29:08,412 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 17:29:09,983 INFO [download_results] running in process 2076424
2024-04-20 17:29:10,003 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 17:29:10,007 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-20 17:29:10,008 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 17:29:10,009 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26oct19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 17:29:14,024 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 17:30:19,530 INFO [download_results] hindcast 2019-10-26 results files from optimum-hindcast downloaded
2024-04-20 17:30:19,558 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 17:30:19,787 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-10-26'], host='localhost')
2024-04-20 17:30:20,962 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 17:30:21,175 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-oct', 'robot.graham'], host='localhost')
2024-04-20 17:30:22,159 INFO [archive_tarball] running in process 2076472
2024-04-20 17:30:22,161 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 17:30:22,162 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-04-20 17:30:22,163 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-20 17:30:22,164 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct19.tar from /ocean/sallen/timbucktwo/oxygen/*oct19/
2024-04-20 17:37:26,769 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct19.index from /ocean/dlatorne/oxygen-oct19.tar
2024-04-20 17:37:26,877 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-oct19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-20 18:11:16,090 INFO [archive_tarball] hindcast *oct19 results files archived to robot.graham
2024-04-20 18:11:16,125 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-20 18:54:18,921 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 18:54:19,343 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-01'], host='localhost')
2024-04-20 18:54:19,349 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 18:54:19,353 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 18:54:21,021 INFO [download_results] running in process 2076955
2024-04-20 18:54:21,022 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 18:54:21,023 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-20 18:54:21,025 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 18:54:21,026 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01nov19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 18:54:25,099 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 18:56:18,467 INFO [download_results] hindcast 2019-11-01 results files from optimum-hindcast downloaded
2024-04-20 18:56:18,482 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 18:56:18,701 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-01'], host='localhost')
2024-04-20 18:56:20,006 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 20:19:29,616 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 20:19:30,035 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-06'], host='localhost')
2024-04-20 20:19:30,040 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 20:19:30,045 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 20:19:31,689 INFO [download_results] running in process 2077339
2024-04-20 20:19:31,691 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 20:19:31,692 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-20 20:19:31,693 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 20:19:31,694 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06nov19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 20:19:35,756 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 20:20:40,407 INFO [download_results] hindcast 2019-11-06 results files from optimum-hindcast downloaded
2024-04-20 20:20:40,445 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 20:20:40,728 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-06'], host='localhost')
2024-04-20 20:20:41,866 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 21:02:09,155 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240421/00/
2024-04-20 21:02:09,156 INFO [collect_weather] 2024-04-21 2.5km weather forecast 00 collection complete
2024-04-20 21:02:09,158 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-20 21:02:09,569 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-04-20 21:02:09,575 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-04-21'], host='localhost')
2024-04-20 21:02:10,603 INFO [collect_weather] running in process 2077487
2024-04-20 21:02:10,604 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 21:02:10,606 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-04-20 21:02:10,608 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-20 21:02:10,622 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-04-20 21:02:12,711 INFO [crop_gribs] running in process 2077489
2024-04-20 21:02:12,712 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 21:02:12,713 INFO [crop_gribs] publishing log messages to tcp://*:5564
2024-04-20 21:02:12,714 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-20 21:02:12,715 INFO [crop_gribs] cropping 2024-04-21 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-04-20 21:02:12,728 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240421/06/
2024-04-20 21:10:32,427 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240421/00/
2024-04-20 21:10:32,428 INFO [crop_gribs] 2024-04-21 00 GRIBs cropping complete
2024-04-20 21:10:32,430 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-20 21:44:40,471 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 21:44:40,934 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-11'], host='localhost')
2024-04-20 21:44:40,941 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 21:44:40,945 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 21:44:42,378 INFO [download_results] running in process 2077940
2024-04-20 21:44:42,379 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 21:44:42,380 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-20 21:44:42,381 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 21:44:42,382 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11nov19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 21:44:46,698 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 21:45:46,941 INFO [download_results] hindcast 2019-11-11 results files from optimum-hindcast downloaded
2024-04-20 21:45:46,956 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 21:45:47,140 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-11'], host='localhost')
2024-04-20 21:45:48,447 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-20 23:09:51,161 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-20 23:09:51,670 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-16'], host='localhost')
2024-04-20 23:09:51,681 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 23:09:51,686 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-20 23:09:53,586 INFO [download_results] running in process 2078534
2024-04-20 23:09:53,587 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-20 23:09:53,588 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-20 23:09:53,589 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-20 23:09:53,590 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16nov19 to /ocean/sallen/timbucktwo/oxygen
2024-04-20 23:09:57,692 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-20 23:10:55,176 INFO [download_results] hindcast 2019-11-16 results files from optimum-hindcast downloaded
2024-04-20 23:10:55,188 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-20 23:10:55,357 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-16'], host='localhost')
2024-04-20 23:10:56,462 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 00:35:01,804 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 00:35:02,220 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-21'], host='localhost')
2024-04-21 00:35:02,225 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 00:35:02,232 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 00:35:03,837 INFO [download_results] running in process 2079043
2024-04-21 00:35:03,838 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 00:35:03,839 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-21 00:35:03,840 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 00:35:03,841 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21nov19 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 00:35:07,913 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 00:36:07,906 INFO [download_results] hindcast 2019-11-21 results files from optimum-hindcast downloaded
2024-04-21 00:36:07,916 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 00:36:08,126 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-21'], host='localhost')
2024-04-21 00:36:09,350 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 02:00:12,450 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 02:00:12,905 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-11-26'], host='localhost')
2024-04-21 02:00:12,912 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 02:00:12,916 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 02:00:14,396 INFO [download_results] running in process 2079348
2024-04-21 02:00:14,397 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 02:00:14,398 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-21 02:00:14,399 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 02:00:14,399 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26nov19 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 02:00:18,568 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 02:01:15,301 INFO [download_results] hindcast 2019-11-26 results files from optimum-hindcast downloaded
2024-04-21 02:01:15,312 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 02:01:15,533 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-11-26'], host='localhost')
2024-04-21 02:01:16,836 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 02:01:17,058 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-nov', 'robot.graham'], host='localhost')
2024-04-21 02:01:18,093 INFO [archive_tarball] running in process 2079396
2024-04-21 02:01:18,094 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 02:01:18,095 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-04-21 02:01:18,096 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-21 02:01:18,097 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-nov19.tar from /ocean/sallen/timbucktwo/oxygen/*nov19/
2024-04-21 02:08:31,180 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-nov19.index from /ocean/dlatorne/oxygen-nov19.tar
2024-04-21 02:08:31,290 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-nov19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-21 02:27:03,921 INFO [archive_tarball] hindcast *nov19 results files archived to robot.graham
2024-04-21 02:27:03,925 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-21 03:10:08,287 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240421/06/
2024-04-21 03:10:08,288 INFO [collect_weather] 2024-04-21 2.5km weather forecast 06 collection complete
2024-04-21 03:10:08,291 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-21 03:10:08,792 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,798 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,803 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,809 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,816 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,820 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,825 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,834 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,837 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,841 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,846 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,857 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,870 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,894 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 03:10:08,902 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-04-21 03:10:08,913 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-04-21 03:10:08,938 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-04-21 03:10:08,948 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-04-21 03:10:08,966 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-04-21 03:10:08,976 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-04-21 03:10:11,480 INFO [collect_NeahBay_ssh] running in process 2079618
2024-04-21 03:10:11,481 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:11,482 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2024-04-21 03:10:11,483 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:11,484 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-04-21
2024-04-21 03:10:11,486 INFO [collect_weather] running in process 2079619
2024-04-21 03:10:11,487 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:11,488 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-04-21 03:10:11,489 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:11,505 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-04-21 03:10:13,469 INFO [collect_NeahBay_ssh] 2024-04-21T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-04-21 03:10:13,475 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-21 03:10:13,831 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 03:10:14,036 INFO [collect_river_data] running in process 2079601
2024-04-21 03:10:14,054 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,085 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-04-21 03:10:14,091 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,093 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-04-20
2024-04-21 03:10:14,094 INFO [collect_river_data] running in process 2079608
2024-04-21 03:10:14,096 INFO [collect_river_data] running in process 2079609
2024-04-21 03:10:14,097 INFO [collect_river_data] running in process 2079599
2024-04-21 03:10:14,098 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,100 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,101 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,101 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-04-21 03:10:14,102 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-04-21 03:10:14,103 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-04-21 03:10:14,104 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,105 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,107 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,110 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-04-20
2024-04-21 03:10:14,112 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-04-20
2024-04-21 03:10:14,113 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-04-20
2024-04-21 03:10:14,114 INFO [collect_river_data] running in process 2079612
2024-04-21 03:10:14,115 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,117 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-21 03:10:14,119 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,119 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-04-20
2024-04-21 03:10:14,120 INFO [collect_river_data] running in process 2079606
2024-04-21 03:10:14,121 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,122 INFO [collect_river_data] publishing log messages to tcp://*:5562
2024-04-21 03:10:14,123 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,124 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-04-20
2024-04-21 03:10:14,146 INFO [collect_river_data] running in process 2079611
2024-04-21 03:10:14,149 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,149 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-04-21 03:10:14,150 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,151 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-04-20
2024-04-21 03:10:14,174 INFO [collect_river_data] running in process 2079605
2024-04-21 03:10:14,175 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,176 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-04-21 03:10:14,177 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,179 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-04-20
2024-04-21 03:10:14,196 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-04-20 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-04-21 03:10:14,197 INFO [collect_river_data] ECCC Capilano river data collection for 2024-04-20 completed
2024-04-21 03:10:14,199 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:14,212 INFO [collect_river_data] running in process 2079610
2024-04-21 03:10:14,213 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,214 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-04-21 03:10:14,215 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,217 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-04-20
2024-04-21 03:10:14,224 INFO [collect_river_data] running in process 2079603
2024-04-21 03:10:14,227 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,229 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-04-21 03:10:14,230 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,231 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-04-20
2024-04-21 03:10:14,232 INFO [collect_river_data] running in process 2079607
2024-04-21 03:10:14,234 INFO [collect_river_data] running in process 2079613
2024-04-21 03:10:14,236 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,237 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,240 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-04-21 03:10:14,241 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-04-21 03:10:14,242 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,244 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,245 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-04-20
2024-04-21 03:10:14,245 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-04-20
2024-04-21 03:10:14,247 INFO [collect_river_data] running in process 2079602
2024-04-21 03:10:14,248 INFO [collect_river_data] running in process 2079614
2024-04-21 03:10:14,249 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,250 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:14,250 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-04-21 03:10:14,252 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-04-21 03:10:14,253 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,254 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:14,255 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-04-20
2024-04-21 03:10:14,255 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-04-20
2024-04-21 03:10:14,259 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-04-21 03:10:14,259 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-04-21 03:10:14,260 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-04-20 completed
2024-04-21 03:10:14,261 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-04-20 completed
2024-04-21 03:10:14,264 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-04-21 03:10:14,265 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-04-21 03:10:14,265 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-04-20 completed
2024-04-21 03:10:14,266 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-04-20 completed
2024-04-21 03:10:14,268 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-04-21 03:10:14,269 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-04-21 03:10:14,270 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-04-20 completed
2024-04-21 03:10:14,270 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-04-20 completed
2024-04-21 03:10:14,275 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-04-20 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-04-21 03:10:14,276 INFO [collect_river_data] ECCC Fraser river data collection for 2024-04-20 completed
2024-04-21 03:10:14,319 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-04-21 03:10:14,320 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-04-20 completed
2024-04-21 03:10:14,326 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-04-21 03:10:14,335 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-04-20 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-04-21 03:10:14,346 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-04-20 completed
2024-04-21 03:10:14,363 INFO [collect_river_data] ECCC Englishman river data collection for 2024-04-20 completed
2024-04-21 03:10:14,381 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-04-21 03:10:14,405 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-04-20 completed
2024-04-21 03:10:14,408 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-04-21 03:10:14,409 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-04-21 03:10:14,410 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-04-20 completed
2024-04-21 03:10:14,411 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-04-20 completed
2024-04-21 03:10:14,541 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:14,762 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:14,993 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:15,122 INFO [crop_gribs] running in process 2079620
2024-04-21 03:10:15,123 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:15,124 INFO [crop_gribs] publishing log messages to tcp://*:5562
2024-04-21 03:10:15,125 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:15,126 INFO [crop_gribs] cropping 2024-04-21 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-04-21 03:10:15,139 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240421/12/
2024-04-21 03:10:15,265 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:15,533 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:15,637 INFO [get_onc_ctd] running in process 2079616
2024-04-21 03:10:15,638 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:15,639 INFO [get_onc_ctd] publishing log messages to tcp://*:5567
2024-04-21 03:10:15,640 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:15,641 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-04-20
2024-04-21 03:10:15,665 INFO [get_onc_ctd] running in process 2079615
2024-04-21 03:10:15,666 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:15,666 INFO [get_onc_ctd] publishing log messages to tcp://*:5568
2024-04-21 03:10:15,667 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:15,668 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-04-20
2024-04-21 03:10:15,669 INFO [get_onc_ferry] running in process 2079617
2024-04-21 03:10:15,670 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:15,671 INFO [get_onc_ferry] publishing log messages to tcp://*:5569
2024-04-21 03:10:15,672 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:15,673 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-04-20
2024-04-21 03:10:15,797 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:16,081 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:16,284 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:16,499 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:16,697 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:16,893 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:17,088 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:17,315 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 03:10:18,451 INFO [make_ssh_files] running in process 2079984
2024-04-21 03:10:18,475 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:18,492 INFO [make_ssh_files] publishing log messages to tcp://*:5566
2024-04-21 03:10:18,510 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:18,511 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-04-21 NOAA Neah Bay 00Z observation and forecast values
2024-04-21 03:10:23,886 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d16.nc
2024-04-21 03:10:24,117 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc
2024-04-21 03:10:24,321 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d18.nc
2024-04-21 03:10:24,539 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d19.nc
2024-04-21 03:10:24,767 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d20.nc
2024-04-21 03:10:25,003 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d21.nc
2024-04-21 03:10:25,201 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d22.nc
2024-04-21 03:10:25,420 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d23.nc
2024-04-21 03:10:25,626 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d24.nc
2024-04-21 03:10:25,916 INFO [make_ssh_files] sea surface height boundary file for 2024-04-21 forecast2 run created
2024-04-21 03:10:25,951 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-21 03:10:26,223 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-21 03:10:26,239 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-21 03:10:27,559 INFO [make_runoff_file] running in process 2080062
2024-04-21 03:10:27,560 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:27,561 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2024-04-21 03:10:27,562 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:27,611 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-21 03:10:27,774 INFO [make_runoff_file] 2024-04-21 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-21 03:10:27,776 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-21 03:10:28,657 INFO [make_v202111_runoff_file] running in process 2080060
2024-04-21 03:10:28,660 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:28,661 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5566
2024-04-21 03:10:28,662 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:28,663 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-20
2024-04-21 03:10:29,852 ERROR [make_v202111_runoff_file] no 2024-04-20 discharge obs for Skagit_MountVernon: patching
2024-04-21 03:10:30,536 ERROR [make_v202111_runoff_file] no 2024-04-20 discharge obs for Snohomish_Monroe: patching
2024-04-21 03:10:30,791 ERROR [make_v202111_runoff_file] no 2024-04-20 discharge obs for Nisqually_McKenna: patching
2024-04-21 03:10:30,898 ERROR [make_v202111_runoff_file] no 2024-04-20 discharge obs for Greenwater_Greenwater: patching
2024-04-21 03:10:32,626 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-20: /results/forcing/rivers/R202108Dailies_y2024m04d20.nc
2024-04-21 03:10:32,627 INFO [make_v202111_runoff_file] 2024-04-20 runoff file creation completed
2024-04-21 03:10:32,629 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-21 03:10:35,271 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-04-20
2024-04-21 03:10:37,427 INFO [get_onc_ctd] 2024-04-20 ONC SEVIP CTD T&S file created
2024-04-21 03:10:37,430 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-04-21 03:10:37,643 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-04-21 03:10:38,749 INFO [ping_erddap] running in process 2080102
2024-04-21 03:10:38,750 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:10:38,752 INFO [ping_erddap] publishing log messages to tcp://*:5566
2024-04-21 03:10:38,753 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:10:38,769 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-04-21 03:10:38,772 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-21 03:10:52,017 CRITICAL [get_onc_ctd] 2024-04-20 ONC SCVIP CTD T&S file creation failed
2024-04-21 03:11:01,554 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-04-20
2024-04-21 03:11:30,656 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-04-20
2024-04-21 03:12:03,625 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-04-20
2024-04-21 03:12:21,036 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-04-20
2024-04-21 03:12:39,413 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-04-20
2024-04-21 03:12:48,370 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-04-20
2024-04-21 03:12:57,567 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-04-20; substituting empty dataset
2024-04-21 03:12:57,567 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-04-20
2024-04-21 03:13:05,933 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-20; substituting empty dataset
2024-04-21 03:14:41,499 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240421/06/
2024-04-21 03:14:41,500 INFO [crop_gribs] 2024-04-21 06 GRIBs cropping complete
2024-04-21 03:14:41,502 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-21 03:14:41,940 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-04-21 03:14:44,321 INFO [grib_to_netcdf] running in process 2080132
2024-04-21 03:14:44,344 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:14:44,362 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2024-04-21 03:14:44,370 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:14:44,371 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-21 forecast2 run
2024-04-21 03:14:58,592 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d22.nc
2024-04-21 03:15:02,507 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d23.nc
2024-04-21 03:15:02,519 INFO [grib_to_netcdf] 2024-04-21 NEMO-atmos forcing file for forecast2 created
2024-04-21 03:15:02,556 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-21 03:15:02,757 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-04-21 03:15:02,787 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-04-21 03:15:02,815 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-04-21 03:15:02,836 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-04-21 03:15:04,014 INFO [upload_forcing] running in process 2080170
2024-04-21 03:15:04,017 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:15:04,018 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-04-21 03:15:04,020 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:15:04,023 INFO [upload_forcing] running in process 2080168
2024-04-21 03:15:04,024 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:15:04,025 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-21 03:15:04,026 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:15:04,028 INFO [upload_forcing] running in process 2080172
2024-04-21 03:15:04,029 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:15:04,030 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-21 03:15:04,031 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:15:04,041 INFO [upload_forcing] running in process 2080171
2024-04-21 03:15:04,042 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:15:04,042 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-21 03:15:04,043 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:15:05,508 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m04d20.nc
2024-04-21 03:15:05,584 INFO [upload_forcing] forecast2 2024-04-21 forcing files upload to optimum-hindcast completed
2024-04-21 03:15:05,588 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 03:15:05,714 INFO [upload_forcing] forecast2 2024-04-21 forcing files upload to orcinus-nowcast-agrif completed
2024-04-21 03:15:05,850 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 03:15:06,631 INFO [upload_forcing] forecast2 2024-04-21 forcing files upload to arbutus.cloud-nowcast completed
2024-04-21 03:15:06,636 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 03:15:06,850 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 03:15:08,160 INFO [make_forcing_links] running in process 2080199
2024-04-21 03:15:08,161 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:15:08,163 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-04-21 03:15:08,164 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:15:09,777 INFO [make_forcing_links] forecast2 2024-04-21 forcing file links on arbutus.cloud-nowcast created
2024-04-21 03:15:09,783 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-21 03:15:10,013 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 03:15:12,139 INFO [run_NEMO] running in process 19143
2024-04-21 03:15:12,160 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:15:12,164 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-21 03:15:12,206 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:15:12,360 INFO [upload_forcing] forecast2 2024-04-21 forcing files upload to robot.graham completed
2024-04-21 03:15:12,405 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 03:15:13,338 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/20apr24forecast2_2024-04-21T031512.276992-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-21 03:15:13,374 INFO [run_NEMO] forecast2 NEMO run for 2024-04-21 on arbutus.cloud-nowcast started
2024-04-21 03:15:13,380 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-21 03:15:13,613 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-21 03:15:15,515 INFO [watch_NEMO] running in process 19400
2024-04-21 03:15:15,517 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:15:15,518 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-21 03:15:15,519 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:15:15,587 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-21 03:15:25,046 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-04-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-04-21 03:15:25,066 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-20 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-04-21 03:15:25,452 INFO [get_onc_ferry] 2024-04-20 ONC TWDP ferry data file created
2024-04-21 03:15:25,454 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-04-21 03:15:25,677 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-04-21 03:15:26,663 INFO [ping_erddap] running in process 2080212
2024-04-21 03:15:26,665 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:15:26,666 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-04-21 03:15:26,667 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:15:26,668 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-04-21 03:15:26,671 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-21 03:20:15,587 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17604463 = 2024-04-22 05:08:00 UTC, 17.1% complete
2024-04-21 03:25:15,587 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17605004 = 2024-04-22 11:08:40 UTC, 37.2% complete
2024-04-21 03:30:15,589 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17605523 = 2024-04-22 16:54:40 UTC, 56.4% complete
2024-04-21 03:30:23,362 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 03:30:23,782 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-12-01'], host='localhost')
2024-04-21 03:30:23,788 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 03:30:23,791 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 03:30:24,963 INFO [download_results] running in process 2080319
2024-04-21 03:30:24,964 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:30:24,966 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-21 03:30:24,966 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:30:24,967 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01dec19 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 03:30:29,114 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 03:31:28,032 INFO [download_results] hindcast 2019-12-01 results files from optimum-hindcast downloaded
2024-04-21 03:31:28,045 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 03:31:28,262 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-12-01'], host='localhost')
2024-04-21 03:31:29,573 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 03:35:15,588 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17606021 = 2024-04-22 22:26:40 UTC, 74.8% complete
2024-04-21 03:40:15,589 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17606509 = 2024-04-23 03:52:00 UTC, 92.9% complete
2024-04-21 03:45:15,593 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-04-21 03:45:15,597 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-21 03:45:16,012 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 03:45:16,025 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 03:45:16,044 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-20'], host='localhost')
2024-04-21 03:45:17,566 INFO [download_results] running in process 2080427
2024-04-21 03:45:17,579 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 03:45:17,583 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-21 03:45:17,584 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 03:45:17,585 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/20apr24 to /results/SalishSea/forecast2.202111
2024-04-21 03:47:13,215 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-21 04:10:02,490 INFO [download_results] forecast2 2024-04-20 results files from arbutus.cloud-nowcast downloaded
2024-04-21 04:10:02,495 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 04:10:02,716 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-04-20'], host='localhost')
2024-04-21 04:10:05,844 INFO [make_CHS_currents_file] running in process 2080490
2024-04-21 04:10:05,845 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:10:05,846 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-04-21 04:10:05,847 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:10:45,160 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-20 for forecast2
2024-04-21 04:10:45,163 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-21 04:10:45,336 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-04-20'], host='localhost')
2024-04-21 04:10:46,484 INFO [update_forecast_datasets] running in process 2080516
2024-04-21 04:10:46,486 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:10:46,496 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-04-21 04:10:46,506 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:10:46,523 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/20apr24 run
2024-04-21 04:15:12,907 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/20apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-21 04:15:12,908 INFO [update_forecast_datasets] nemo 2024-04-20 forecast2 rolling forecast datasets updated
2024-04-21 04:15:12,912 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-21 04:15:13,136 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-21 04:15:13,141 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-04-20'], host='localhost')
2024-04-21 04:15:13,146 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-04-20'], host='localhost')
2024-04-21 04:15:14,349 INFO [ping_erddap] running in process 2080589
2024-04-21 04:15:14,376 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:15:14,394 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-04-21 04:15:14,409 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:15:14,420 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-21 04:15:14,422 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-21 04:15:16,260 INFO [make_surface_current_tiles] running in process 2080592
2024-04-21 04:15:16,263 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:15:16,264 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-04-21 04:15:16,265 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:15:16,312 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-20 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/20apr24
2024-04-21 04:15:20,761 INFO [make_plots] running in process 2080591
2024-04-21 04:15:20,762 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:15:20,763 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-21 04:15:20,765 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:15:21,919 INFO [make_plots] preparing render list for 2024-04-20 NEMO forecast2 publish figures
2024-04-21 04:15:22,127 INFO [make_plots] added Website_thumbnail figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,128 INFO [make_plots] added Threshold_website figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,129 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,130 INFO [make_plots] added Boundary Bay figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,131 INFO [make_plots] added Campbell River figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,132 INFO [make_plots] added Cherry Point figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,133 INFO [make_plots] added Friday Harbor figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,134 INFO [make_plots] added Halfmoon Bay figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,135 INFO [make_plots] added Nanaimo figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,136 INFO [make_plots] added Neah Bay figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,137 INFO [make_plots] added New Westminster figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,137 INFO [make_plots] added Patricia Bay figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,139 INFO [make_plots] added Point Atkinson figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,140 INFO [make_plots] added Port Renfrew figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,141 INFO [make_plots] added Sand Heads figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,141 INFO [make_plots] added Sandy Cove figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,142 INFO [make_plots] added Squamish figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,143 INFO [make_plots] added Victoria figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,144 INFO [make_plots] added Woodwards Landing figure to 2024-04-20 NEMO forecast2 publish render list
2024-04-21 04:15:22,145 INFO [make_plots] starting to render nemo forecast2 publish 20apr24 figures
2024-04-21 04:31:26,483 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/20apr24
2024-04-21 04:35:13,953 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-21 04:35:13,954 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-21 04:35:52,782 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:36:01,928 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/20apr24
2024-04-21 04:36:01,937 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-20 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/20apr24
2024-04-21 04:36:01,938 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-20 forecast2 completed
2024-04-21 04:36:02,234 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-21 04:37:48,500 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:37:48,907 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:38:26,208 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:39:42,637 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:40:21,658 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:41:00,561 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:41:38,500 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:42:16,401 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:42:54,901 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:43:33,748 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:43:34,181 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:44:12,258 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:44:50,398 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-16 00:00:00Z to 2024-04-23 05:55:00Z
2024-04-21 04:45:29,365 INFO [make_plots] finished rendering nemo forecast2 publish 20apr24 figures
2024-04-21 04:45:29,367 INFO [make_plots] nemo publish plots for 2024-04-20 forecast2 completed
2024-04-21 04:45:29,376 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-21 04:45:29,809 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-04-20'], host='localhost')
2024-04-21 04:45:34,227 INFO [make_feeds] running in process 2082827
2024-04-21 04:45:34,228 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:45:34,229 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-04-21 04:45:34,230 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:46:03,093 INFO [make_feeds] ATOM feeds for 2024-04-20 forecast2 run completed
2024-04-21 04:46:03,097 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-21 04:46:03,495 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-04-21 04:46:04,608 INFO [clear_checklist] running in process 2082851
2024-04-21 04:46:04,610 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:46:04,611 INFO [clear_checklist] publishing log messages to tcp://*:5565
2024-04-21 04:46:04,612 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:46:04,613 INFO [clear_checklist] requesting that manager clear system state checklist
2024-04-21 04:46:04,616 INFO [manager] writing checklist to log file
2024-04-21 04:46:04,709 INFO [manager] checklist cleared
2024-04-21 04:46:04,731 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-04-21 04:46:04,733 INFO [clear_checklist] nowcast system checklist cleared
2024-04-21 04:46:05,815 INFO [rotate_logs] running in process 2082856
2024-04-21 04:46:05,816 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:46:05,818 INFO [rotate_logs] publishing log messages to tcp://*:5565
2024-04-21 04:46:05,819 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:46:05,819 INFO [rotate_logs] rotating log files