2024-07-22 04:26:14,868 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-07-22 04:26:14,874 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-07-22 04:26:14,878 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-07-22 04:26:14,880 INFO [rotate_logs] log files rotated
2024-07-22 04:26:14,885 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-07-22 09:11:43,536 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240722/12/
2024-07-22 09:11:43,538 INFO [collect_weather] 2024-07-22 2.5km weather forecast 12 collection complete
2024-07-22 09:11:43,541 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-22 09:11:43,713 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 09:11:43,719 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 09:11:43,724 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 09:11:43,728 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-07-21'], host='localhost')
2024-07-22 09:11:43,732 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-07-22 09:11:43,737 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-07-22 09:11:43,741 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-07-22 09:11:43,744 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-07-22 09:11:43,748 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-07-22 09:11:45,124 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240722/12/
2024-07-22 09:11:45,124 INFO [crop_gribs] 2024-07-22 12 GRIBs cropping complete
2024-07-22 09:11:45,126 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-22 09:11:45,270 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-07-22 09:11:45,377 INFO [collect_weather] running in process 727544
2024-07-22 09:11:45,378 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:45,379 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-22 09:11:45,383 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:45,401 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-07-22 09:11:45,413 INFO [collect_NeahBay_ssh] running in process 727542
2024-07-22 09:11:45,414 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:45,415 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-07-22 09:11:45,416 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:45,417 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-07-22
2024-07-22 09:11:45,457 INFO [download_live_ocean] running in process 727543
2024-07-22 09:11:45,459 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:45,460 INFO [download_live_ocean] publishing log messages to tcp://*:5565
2024-07-22 09:11:45,461 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:45,463 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-07-22
2024-07-22 09:11:45,940 INFO [collect_river_data] running in process 727536
2024-07-22 09:11:45,941 INFO [collect_river_data] running in process 727540
2024-07-22 09:11:45,942 INFO [collect_river_data] running in process 727539
2024-07-22 09:11:45,943 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:45,944 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:45,945 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:45,946 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-07-22 09:11:45,947 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-22 09:11:45,948 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-22 09:11:45,949 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:45,949 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:45,950 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:45,951 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-07-21
2024-07-22 09:11:45,952 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-07-21
2024-07-22 09:11:45,953 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-07-21
2024-07-22 09:11:45,956 INFO [make_turbidity_file] running in process 727541
2024-07-22 09:11:45,957 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:45,959 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2024-07-22 09:11:45,960 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:45,961 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-07-22
2024-07-22 09:11:46,210 INFO [collect_river_data] running in process 727538
2024-07-22 09:11:46,211 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:46,212 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-22 09:11:46,213 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:46,214 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-07-21
2024-07-22 09:11:46,275 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-07-21&endDT=2024-07-21 "HTTP/1.1 200 OK"
2024-07-22 09:11:46,280 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-07-22 09:11:46,281 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-07-21 completed
2024-07-22 09:11:46,282 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-07-21&endDT=2024-07-21 "HTTP/1.1 200 OK"
2024-07-22 09:11:46,283 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-07-21&endDT=2024-07-21 "HTTP/1.1 200 OK"
2024-07-22 09:11:46,285 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 09:11:46,289 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 09:11:46,293 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 09:11:46,295 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-07-22 09:11:46,296 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-07-22 09:11:46,297 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-07-21 completed
2024-07-22 09:11:46,297 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-07-21 completed
2024-07-22 09:11:46,465 INFO [crop_gribs] running in process 727546
2024-07-22 09:11:46,466 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:46,467 INFO [crop_gribs] publishing log messages to tcp://*:5571
2024-07-22 09:11:46,469 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:46,470 INFO [crop_gribs] cropping 2024-07-22 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-07-22 09:11:46,487 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240722/18/
2024-07-22 09:11:46,564 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-07-21&endDT=2024-07-21 "HTTP/1.1 200 OK"
2024-07-22 09:11:46,569 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-07-21 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-07-22 09:11:46,570 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-07-21 completed
2024-07-22 09:11:46,572 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-22 09:11:46,620 INFO [collect_NeahBay_ssh] 2024-07-22T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-07-22 09:11:46,622 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-22 09:11:46,628 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 09:11:46,656 INFO [download_live_ocean] downloaded 10119612 bytes from https://liveocean.apl.uw.edu/output/f2024.07.22/ubc.nc to /results/forcing/LiveOcean/downloaded/20240722/low_passed_UBC.nc
2024-07-22 09:11:46,657 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-07-22 09:11:47,583 INFO [grib_to_netcdf] running in process 727666
2024-07-22 09:11:47,586 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:47,587 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2024-07-22 09:11:47,588 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:47,589 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-22 nowcast and forecast runs
2024-07-22 09:11:47,693 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240722/low_passed_UBC.nc
2024-07-22 09:11:47,694 INFO [download_live_ocean] 2024-07-22 Live Ocean file for Salish Sea western boundary downloaded
2024-07-22 09:11:47,696 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-07-22 09:11:47,849 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-07-22'], host='localhost')
2024-07-22 09:11:49,021 INFO [make_turbidity_file] 2024-07-22 Fraser River turbidity file creation complete
2024-07-22 09:11:49,023 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-07-22 09:11:49,188 INFO [make_ssh_files] running in process 727714
2024-07-22 09:11:49,189 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:49,190 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2024-07-22 09:11:49,191 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:49,192 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-07-22 NOAA Neah Bay 06Z observation and forecast values
2024-07-22 09:11:49,840 INFO [make_live_ocean_files] running in process 727741
2024-07-22 09:11:49,841 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:49,842 INFO [make_live_ocean_files] publishing log messages to tcp://*:5566
2024-07-22 09:11:49,843 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:49,851 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-07-22 Live Ocean run
2024-07-22 09:11:53,460 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc
2024-07-22 09:11:53,637 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc
2024-07-22 09:11:53,818 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-22 09:11:53,999 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-22 09:11:54,391 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d22.nc
2024-07-22 09:11:54,574 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d23.nc
2024-07-22 09:11:54,756 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d24.nc
2024-07-22 09:11:54,940 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d25.nc
2024-07-22 09:11:55,051 INFO [make_ssh_files] sea surface height boundary file for 2024-07-22 nowcast run created
2024-07-22 09:11:55,054 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-22 09:11:55,060 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-22 09:11:55,064 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-22 09:11:56,051 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m07d22.nc
2024-07-22 09:11:56,392 INFO [make_runoff_file] running in process 727790
2024-07-22 09:11:56,393 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:56,393 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2024-07-22 09:11:56,394 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:56,424 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-22 09:11:56,480 INFO [make_runoff_file] 2024-07-22 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-22 09:11:56,482 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-22 09:11:56,979 INFO [make_v202111_runoff_file] running in process 727788
2024-07-22 09:11:56,980 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:11:56,981 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5565
2024-07-22 09:11:56,982 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:11:56,983 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-21
2024-07-22 09:11:59,505 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-21: /results/forcing/rivers/R202108Dailies_y2024m07d21.nc
2024-07-22 09:11:59,506 INFO [make_v202111_runoff_file] 2024-07-21 runoff file creation completed
2024-07-22 09:11:59,507 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-22 09:12:03,054 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d23.nc
2024-07-22 09:12:06,784 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d24.nc
2024-07-22 09:12:06,796 INFO [grib_to_netcdf] 2024-07-22 NEMO-atmos forcing file for nowcast+ created
2024-07-22 09:12:06,799 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-22 09:12:29,620 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d22.nc
2024-07-22 09:12:29,622 INFO [make_live_ocean_files] 2024-07-22 Live Ocean western boundary conditions files created
2024-07-22 09:12:29,624 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-07-22 09:12:29,631 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-07-22 09:12:29,637 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-07-22 09:12:29,639 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-07-22 09:12:29,644 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-07-22 09:12:29,649 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-07-22 09:12:30,714 INFO [ping_erddap] running in process 727834
2024-07-22 09:12:30,715 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:12:30,717 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-22 09:12:30,718 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:12:30,720 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-07-22 09:12:30,722 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 09:12:30,814 INFO [upload_forcing] running in process 727838
2024-07-22 09:12:30,815 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:12:30,816 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-07-22 09:12:30,817 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:12:30,825 INFO [upload_forcing] running in process 727836
2024-07-22 09:12:30,826 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:12:30,827 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-22 09:12:30,828 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:12:30,835 INFO [upload_forcing] running in process 727837
2024-07-22 09:12:30,837 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:12:30,838 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-22 09:12:30,839 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:12:30,878 INFO [upload_forcing] running in process 727839
2024-07-22 09:12:30,879 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:12:30,881 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-22 09:12:30,882 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:12:33,238 INFO [upload_forcing] nowcast+ 2024-07-22 forcing files upload to optimum-hindcast completed
2024-07-22 09:12:33,242 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 09:12:33,574 INFO [upload_forcing] nowcast+ 2024-07-22 forcing files upload to orcinus-nowcast-agrif completed
2024-07-22 09:12:33,578 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 09:12:40,390 INFO [upload_forcing] nowcast+ 2024-07-22 forcing files upload to robot.graham completed
2024-07-22 09:12:40,395 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 09:12:42,923 INFO [upload_forcing] nowcast+ 2024-07-22 forcing files upload to arbutus.cloud-nowcast completed
2024-07-22 09:12:42,928 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 09:12:42,936 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 09:12:44,305 INFO [make_forcing_links] running in process 727873
2024-07-22 09:12:44,306 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:12:44,307 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-22 09:12:44,309 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:12:45,432 INFO [make_forcing_links] nowcast+ 2024-07-22 forcing file links on arbutus.cloud-nowcast created
2024-07-22 09:12:45,436 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-22 09:12:45,448 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 09:12:47,685 INFO [run_NEMO] running in process 17720
2024-07-22 09:12:47,687 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:12:47,688 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-22 09:12:47,697 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:12:48,593 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/22jul24nowcast_2024-07-22T091247.737419-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-22 09:12:48,612 INFO [run_NEMO] nowcast NEMO run for 2024-07-22 on arbutus.cloud-nowcast started
2024-07-22 09:12:48,617 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-22 09:12:48,627 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-22 09:12:50,621 INFO [watch_NEMO] running in process 17974
2024-07-22 09:12:50,622 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:12:50,625 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-22 09:12:50,627 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:12:50,693 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-22 09:17:50,694 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17800985 = 2024-07-22 04:42:40 UTC, 19.6% complete
2024-07-22 09:22:50,695 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17801474 = 2024-07-22 10:08:40 UTC, 42.3% complete
2024-07-22 09:27:50,696 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17801940 = 2024-07-22 15:19:20 UTC, 63.9% complete
2024-07-22 09:32:50,697 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17802439 = 2024-07-22 20:52:00 UTC, 87.0% complete
2024-07-22 09:37:50,701 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-07-22 09:37:50,706 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-22 09:37:50,898 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 09:37:50,904 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 09:37:52,149 INFO [download_results] running in process 727950
2024-07-22 09:37:52,150 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:37:52,152 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-22 09:37:52,153 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:37:52,154 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/22jul24 to /results/SalishSea/nowcast-blue.202111
2024-07-22 09:37:52,156 INFO [make_forcing_links] running in process 727949
2024-07-22 09:37:52,157 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:37:52,158 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-07-22 09:37:52,159 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:37:53,090 INFO [make_forcing_links] ssh 2024-07-22 forcing file links on arbutus.cloud-nowcast created
2024-07-22 09:37:53,094 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-22 09:37:53,105 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 09:37:56,332 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/22jul24forecast_2024-07-22T093755.451882-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-22 09:37:56,348 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-22 09:37:56,351 INFO [run_NEMO] forecast NEMO run for 2024-07-22 on arbutus.cloud-nowcast started
2024-07-22 09:37:56,360 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-22 09:37:59,317 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-22 09:42:59,924 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17803153 = 2024-07-23 04:48:00 UTC, 13.3% complete
2024-07-22 09:47:59,934 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17803634 = 2024-07-23 10:08:40 UTC, 28.2% complete
2024-07-22 09:52:59,957 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17804087 = 2024-07-23 15:10:40 UTC, 42.2% complete
2024-07-22 09:57:59,932 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17804542 = 2024-07-23 20:14:00 UTC, 56.2% complete
2024-07-22 09:58:12,331 INFO [download_results] nowcast 2024-07-22 results files from arbutus.cloud-nowcast downloaded
2024-07-22 09:58:12,336 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 09:58:12,348 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 09:58:12,353 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-07-21'], host='localhost')
2024-07-22 09:58:12,358 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 09:58:14,366 INFO [make_CHS_currents_file] running in process 728125
2024-07-22 09:58:14,367 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:58:14,368 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-22 09:58:14,369 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:58:15,636 INFO [make_plots] running in process 728124
2024-07-22 09:58:15,637 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:58:15,638 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-22 09:58:15,639 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:58:15,650 INFO [make_plots] running in process 728122
2024-07-22 09:58:15,680 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 09:58:15,687 INFO [make_plots] publishing log messages to tcp://*:5566
2024-07-22 09:58:15,688 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-22 09:58:16,304 INFO [make_plots] preparing render list for 2024-07-22 NEMO nowcast-blue research figures
2024-07-22 09:58:16,306 INFO [make_plots] preparing render list for 2024-07-21 NEMO nowcast-blue comparison figures
2024-07-22 09:58:17,390 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-blue research render list
2024-07-22 09:58:18,096 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-blue research render list
2024-07-22 09:58:18,096 INFO [make_plots] added Currents_sections_and_surface figure to 2024-07-22 NEMO nowcast-blue research render list
2024-07-22 09:58:18,097 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-07-22 NEMO nowcast-blue research render list
2024-07-22 09:58:18,099 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-07-22 NEMO nowcast-blue research render list
2024-07-22 09:58:18,100 INFO [make_plots] starting to render nemo nowcast research 22jul24 figures
2024-07-22 09:58:34,747 INFO [make_plots] added SH_wind figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,748 INFO [make_plots] added Compare_VENUS_East figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,749 INFO [make_plots] added Compare_VENUS_Central figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,750 INFO [make_plots] added Central_ADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,751 INFO [make_plots] added Central_depavADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,752 INFO [make_plots] added Central_timeavADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,752 INFO [make_plots] added East_ADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,753 INFO [make_plots] added East_depavADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,754 INFO [make_plots] added East_timeavADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,755 INFO [make_plots] added ddl_ADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,756 INFO [make_plots] added ddl_depavADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,756 INFO [make_plots] added ddl_timeavADCP figure to 2024-07-21 NEMO nowcast-blue comparison render list
2024-07-22 09:58:34,757 INFO [make_plots] starting to render nemo nowcast comparison 21jul24 figures
2024-07-22 09:58:45,443 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-22 for nowcast
2024-07-22 09:58:45,446 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-22 09:59:28,275 INFO [make_plots] finished rendering nemo nowcast research 22jul24 figures
2024-07-22 09:59:28,277 INFO [make_plots] nemo research plots for 2024-07-22 nowcast completed
2024-07-22 09:59:28,296 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-22 10:00:11,178 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available
2024-07-22 10:00:11,739 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-22 10:00:12,229 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-22 10:00:12,717 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-22 10:00:13,211 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-22 10:00:13,703 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-22 10:00:14,196 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-22 10:00:14,684 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-22 10:00:15,175 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-22 10:00:15,668 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-22 10:00:15,669 INFO [make_plots] finished rendering nemo nowcast comparison 21jul24 figures
2024-07-22 10:00:15,759 INFO [make_plots] nemo comparison plots for 2024-07-21 nowcast completed
2024-07-22 10:00:15,763 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-22 10:02:59,267 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17805028 = 2024-07-24 01:38:00 UTC, 71.2% complete
2024-07-22 10:07:59,268 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17805497 = 2024-07-24 06:50:40 UTC, 85.7% complete
2024-07-22 10:12:59,269 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17805960 = 2024-07-24 11:59:20 UTC, 100.0% complete
2024-07-22 10:17:59,273 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-07-22 10:17:59,277 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-22 10:17:59,435 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 10:17:59,443 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 10:17:59,447 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:17:59,454 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:17:59,459 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:17:59,465 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:17:59,469 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:18:00,741 INFO [upload_forcing] running in process 728381
2024-07-22 10:18:00,742 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:00,743 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-07-22 10:18:00,745 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:00,746 INFO [download_results] running in process 728386
2024-07-22 10:18:00,747 INFO [upload_forcing] running in process 728385
2024-07-22 10:18:00,748 INFO [upload_forcing] running in process 728383
2024-07-22 10:18:00,749 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:00,751 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:00,752 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:00,753 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-22 10:18:00,754 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-22 10:18:00,755 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-22 10:18:00,756 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:00,757 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:00,758 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:00,759 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/22jul24 to /results/SalishSea/forecast.202111
2024-07-22 10:18:00,760 INFO [upload_forcing] running in process 728384
2024-07-22 10:18:00,761 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:00,761 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-22 10:18:00,762 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:01,399 INFO [upload_forcing] turbidity 2024-07-22 forcing files upload to optimum-hindcast completed
2024-07-22 10:18:01,404 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 10:18:01,508 INFO [upload_forcing] turbidity 2024-07-22 forcing files upload to orcinus-nowcast-agrif completed
2024-07-22 10:18:01,512 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 10:18:01,528 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:18:01,777 INFO [upload_forcing] turbidity 2024-07-22 forcing files upload to arbutus.cloud-nowcast completed
2024-07-22 10:18:01,783 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 10:18:01,796 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:18:02,416 INFO [make_ww3_wind_file] running in process 18981
2024-07-22 10:18:02,417 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:02,418 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-22 10:18:02,457 INFO [make_ww3_current_file] running in process 18993
2024-07-22 10:18:02,459 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:02,460 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-22 10:18:02,485 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:02,486 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-22 forecast run
2024-07-22 10:18:02,490 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:02,490 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-22 forecast run
2024-07-22 10:18:02,728 INFO [make_forcing_links] running in process 728417
2024-07-22 10:18:02,729 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:02,730 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-22 10:18:02,731 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:03,055 INFO [make_forcing_links] running in process 728419
2024-07-22 10:18:03,056 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:03,057 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2024-07-22 10:18:03,058 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:03,561 INFO [make_forcing_links] nowcast-agrif 2024-07-22 forcing file links on orcinus-nowcast-agrif created
2024-07-22 10:18:03,566 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-22 10:18:03,577 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:18:04,848 INFO [run_NEMO_agrif] running in process 728433
2024-07-22 10:18:04,849 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:04,850 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2024-07-22 10:18:04,851 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:05,103 INFO [upload_forcing] turbidity 2024-07-22 forcing files upload to robot.graham completed
2024-07-22 10:18:05,108 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-22 10:18:07,529 INFO [make_forcing_links] nowcast-green 2024-07-22 forcing file links on arbutus.cloud-nowcast created
2024-07-22 10:18:07,534 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-22 10:18:07,549 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 10:18:12,751 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/22jul24nowcast-green_2024-07-22T101811.480116-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-22 10:18:12,774 INFO [run_NEMO] nowcast-green NEMO run for 2024-07-22 on arbutus.cloud-nowcast started
2024-07-22 10:18:12,780 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-22 10:18:12,798 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-07-22 10:18:13,037 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-22 forecast run
2024-07-22 10:18:13,046 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-22 10:18:16,685 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-22 10:18:17,366 INFO [run_NEMO_agrif] job id for 22jul24nowcast-agrif: 11101965.orca2.ibb
2024-07-22 10:18:17,367 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-07-22 queued on orcinus-nowcast-agrif
2024-07-22 10:18:17,372 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-07-22 10:18:17,387 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11101965.orca2.ibb'], host='localhost')
2024-07-22 10:18:18,756 INFO [watch_NEMO_agrif] running in process 728451
2024-07-22 10:18:18,757 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:18:18,758 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2024-07-22 10:18:18,759 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:18:19,543 INFO [watch_NEMO_agrif] watching 22jul24nowcast-agrif job 11101965 on orcinus-nowcast-agrif
2024-07-22 10:18:19,775 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif job 11101965 is queued on orcinus-nowcast-agrif
2024-07-22 10:19:02,116 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-22 forecast run
2024-07-22 10:19:02,118 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-22 10:19:02,137 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 10:19:13,234 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-22 10:19:13,254 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-22 10:19:20,812 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-07-22 10:19:24,868 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-22 10:23:17,208 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17800731 = 2024-07-22 01:53:20 UTC, 7.9% complete
2024-07-22 10:24:21,250 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7821551 = 2024-07-22 02:06:40 UTC, 8.8% complete
2024-07-22 10:24:24,855 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete
2024-07-22 10:28:17,179 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17800924 = 2024-07-22 04:02:00 UTC, 16.8% complete
2024-07-22 10:29:21,724 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7821751 = 2024-07-22 04:20:00 UTC, 18.1% complete
2024-07-22 10:29:24,875 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 183, 42.4% complete
2024-07-22 10:33:17,212 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17801116 = 2024-07-22 06:10:00 UTC, 25.7% complete
2024-07-22 10:34:22,193 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7821952 = 2024-07-22 06:34:00 UTC, 27.4% complete
2024-07-22 10:34:24,844 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 282, 65.3% complete
2024-07-22 10:38:17,177 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17801302 = 2024-07-22 08:14:00 UTC, 34.3% complete
2024-07-22 10:39:22,664 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7822155 = 2024-07-22 08:49:20 UTC, 36.8% complete
2024-07-22 10:39:24,881 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 378, 87.5% complete
2024-07-22 10:43:17,234 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17801488 = 2024-07-22 10:18:00 UTC, 42.9% complete
2024-07-22 10:44:23,101 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7822355 = 2024-07-22 11:02:40 UTC, 46.0% complete
2024-07-22 10:44:24,813 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-22 10:44:24,816 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-22 10:44:24,983 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:44:24,989 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-22'], host='arbutus.cloud-nowcast')
2024-07-22 10:44:26,336 INFO [download_wwatch3_results] running in process 728712
2024-07-22 10:44:26,338 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:44:26,339 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-07-22 10:44:26,340 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:44:36,164 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-22 10:44:36,183 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-22 10:44:47,973 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-22 10:48:17,331 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17801674 = 2024-07-22 12:22:00 UTC, 51.6% complete
2024-07-22 10:49:16,087 INFO [download_results] forecast 2024-07-22 results files from arbutus.cloud-nowcast downloaded
2024-07-22 10:49:16,091 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 10:49:16,109 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:49:18,101 INFO [make_CHS_currents_file] running in process 728781
2024-07-22 10:49:18,103 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:49:18,104 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-07-22 10:49:18,105 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:49:23,541 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7822556 = 2024-07-22 13:16:40 UTC, 55.3% complete
2024-07-22 10:49:47,391 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete
2024-07-22 10:49:52,052 INFO [download_wwatch3_results] nowcast 2024-07-22 results files from arbutus.cloud-nowcast downloaded
2024-07-22 10:49:52,055 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-22 10:50:01,125 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-22 for forecast
2024-07-22 10:50:01,128 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-22 10:50:01,148 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:50:02,174 INFO [update_forecast_datasets] running in process 728808
2024-07-22 10:50:02,175 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:50:02,176 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-07-22 10:50:02,177 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:50:02,178 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/22jul24 run
2024-07-22 10:50:02,436 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/22jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-22 10:50:02,437 INFO [update_forecast_datasets] nemo 2024-07-22 forecast rolling forecast datasets updated
2024-07-22 10:50:02,439 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-22 10:50:02,457 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-22 10:50:02,461 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:50:02,465 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 10:50:03,618 INFO [ping_erddap] running in process 728813
2024-07-22 10:50:03,619 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:50:03,620 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-22 10:50:03,621 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:50:03,630 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-22 10:50:03,632 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 10:50:04,653 INFO [make_surface_current_tiles] running in process 728816
2024-07-22 10:50:04,654 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:50:04,655 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-07-22 10:50:04,656 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:50:04,659 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-22 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/22jul24
2024-07-22 10:50:06,521 INFO [make_plots] running in process 728815
2024-07-22 10:50:06,522 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 10:50:06,523 INFO [make_plots] publishing log messages to tcp://*:5566
2024-07-22 10:50:06,525 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-22 10:50:07,068 INFO [make_plots] preparing render list for 2024-07-22 NEMO forecast publish figures
2024-07-22 10:50:07,191 INFO [make_plots] added Website_thumbnail figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,192 INFO [make_plots] added Threshold_website figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,193 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,194 INFO [make_plots] added Boundary Bay figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,195 INFO [make_plots] added Campbell River figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,195 INFO [make_plots] added Cherry Point figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,196 INFO [make_plots] added Friday Harbor figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,197 INFO [make_plots] added Halfmoon Bay figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,197 INFO [make_plots] added Nanaimo figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,198 INFO [make_plots] added Neah Bay figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,199 INFO [make_plots] added New Westminster figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,199 INFO [make_plots] added Patricia Bay figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,200 INFO [make_plots] added Point Atkinson figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,200 INFO [make_plots] added Port Renfrew figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,201 INFO [make_plots] added Sand Heads figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,202 INFO [make_plots] added Sandy Cove figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,202 INFO [make_plots] added Squamish figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,203 INFO [make_plots] added Victoria figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,203 INFO [make_plots] added Woodwards Landing figure to 2024-07-22 NEMO forecast publish render list
2024-07-22 10:50:07,204 INFO [make_plots] starting to render nemo forecast publish 22jul24 figures
2024-07-22 10:53:16,544 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17801861 = 2024-07-22 14:26:40 UTC, 60.2% complete
2024-07-22 10:54:24,017 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7822759 = 2024-07-22 15:32:00 UTC, 64.8% complete
2024-07-22 10:54:47,324 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 183, 28.2% complete
2024-07-22 10:58:16,545 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17802049 = 2024-07-22 16:32:00 UTC, 68.9% complete
2024-07-22 10:59:24,451 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7822961 = 2024-07-22 17:46:40 UTC, 74.1% complete
2024-07-22 10:59:47,325 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 282, 43.5% complete
2024-07-22 11:00:36,683 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/22jul24
2024-07-22 11:03:09,290 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-22 11:03:09,292 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-22 11:03:16,547 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17802240 = 2024-07-22 18:39:20 UTC, 77.8% complete
2024-07-22 11:03:44,870 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:04:24,883 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7823161 = 2024-07-22 20:00:00 UTC, 83.4% complete
2024-07-22 11:04:47,326 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 381, 58.8% complete
2024-07-22 11:05:06,723 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/22jul24
2024-07-22 11:05:06,735 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-22 forecast into /results/nowcast-sys/figures/surface_currents/forecast/22jul24
2024-07-22 11:05:06,736 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-22 forecast completed
2024-07-22 11:05:07,050 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-22 11:05:32,240 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:05:32,602 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:06:07,686 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:07:16,798 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:07:52,211 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:08:16,548 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17802422 = 2024-07-22 20:40:40 UTC, 86.2% complete
2024-07-22 11:08:27,985 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:09:03,698 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:09:25,352 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7823363 = 2024-07-22 22:14:40 UTC, 92.7% complete
2024-07-22 11:09:38,034 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:09:47,328 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 471, 72.7% complete
2024-07-22 11:10:14,006 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:10:49,721 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:10:50,074 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:11:25,382 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:11:59,323 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-17 00:00:00Z to 2024-07-24 11:55:00Z
2024-07-22 11:12:34,711 INFO [make_plots] finished rendering nemo forecast publish 22jul24 figures
2024-07-22 11:12:34,714 INFO [make_plots] nemo publish plots for 2024-07-22 forecast completed
2024-07-22 11:12:34,721 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-22 11:12:34,856 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 11:12:37,983 INFO [make_feeds] running in process 730709
2024-07-22 11:12:37,986 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 11:12:37,988 INFO [make_feeds] publishing log messages to tcp://*:5565
2024-07-22 11:12:37,988 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-22 11:13:06,324 INFO [make_feeds] ATOM feeds for 2024-07-22 forecast run completed
2024-07-22 11:13:06,328 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-22 11:13:16,549 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17802606 = 2024-07-22 22:43:20 UTC, 94.7% complete
2024-07-22 11:14:25,775 INFO [watch_NEMO_agrif] 22jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7823520 = 2024-07-22 23:59:20 UTC, 100.0% complete
2024-07-22 11:14:47,328 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 567, 87.5% complete
2024-07-22 11:18:16,550 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17802720 = 2024-07-22 23:59:20 UTC, 100.0% complete
2024-07-22 11:19:25,991 INFO [watch_NEMO_agrif] NEMO AGRIF run 11101965.orca2.ibb on orcinus-nowcast-agrif completed
2024-07-22 11:19:25,995 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-07-22 11:19:26,315 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 11:19:27,830 INFO [download_results] running in process 730758
2024-07-22 11:19:27,831 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 11:19:27,832 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-22 11:19:27,833 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 11:19:27,834 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/22jul24 to /results/SalishSea/nowcast-agrif.201702
2024-07-22 11:19:47,329 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-22 11:19:50,989 INFO [download_results] nowcast-agrif 2024-07-22 results files from orcinus-nowcast-agrif downloaded
2024-07-22 11:19:50,997 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 11:19:51,132 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 11:19:54,480 INFO [make_plots] running in process 730766
2024-07-22 11:19:54,482 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 11:19:54,483 INFO [make_plots] publishing log messages to tcp://*:5564
2024-07-22 11:19:54,484 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-22 11:19:54,895 INFO [make_plots] preparing render list for 2024-07-22 NEMO nowcast-agrif research figures
2024-07-22 11:19:54,896 INFO [make_plots] added baynes_sound_surface figure to 2024-07-22 NEMO nowcast-agrif research render list
2024-07-22 11:19:54,897 INFO [make_plots] starting to render nemo nowcast-agrif research 22jul24 figures
2024-07-22 11:20:05,189 INFO [make_plots] finished rendering nemo nowcast-agrif research 22jul24 figures
2024-07-22 11:20:05,192 INFO [make_plots] nemo research plots for 2024-07-22 nowcast-agrif completed
2024-07-22 11:20:05,196 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-22 11:23:16,785 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-07-22 11:23:16,789 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-22 11:23:17,107 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 11:23:18,644 INFO [download_results] running in process 730809
2024-07-22 11:23:18,645 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 11:23:18,646 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-22 11:23:18,648 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 11:23:18,649 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/22jul24 to /results2/SalishSea/nowcast-green.202111
2024-07-22 11:24:47,538 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-22 11:24:47,542 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-22 11:24:47,854 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 11:24:49,166 INFO [download_wwatch3_results] running in process 730826
2024-07-22 11:24:49,167 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 11:24:49,169 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-07-22 11:24:49,170 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 11:30:22,903 INFO [download_wwatch3_results] forecast 2024-07-22 results files from arbutus.cloud-nowcast downloaded
2024-07-22 11:30:22,907 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-22 11:30:23,079 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 11:30:24,404 INFO [update_forecast_datasets] running in process 730865
2024-07-22 11:30:24,405 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 11:30:24,406 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-07-22 11:30:24,407 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-22 11:30:24,409 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/22jul24 run
2024-07-22 11:30:24,411 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/22jul24 run
2024-07-22 11:30:24,453 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/22jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-22 11:30:24,454 INFO [update_forecast_datasets] wwatch3 2024-07-22 forecast rolling forecast datasets updated
2024-07-22 11:30:24,457 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-22 11:30:24,624 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-22 11:30:25,830 INFO [ping_erddap] running in process 730870
2024-07-22 11:30:25,831 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 11:30:25,833 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-07-22 11:30:25,834 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 11:30:25,836 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-22 11:30:25,838 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 11:30:26,009 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 11:30:29,604 INFO [make_plots] running in process 730875
2024-07-22 11:30:29,606 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 11:30:29,607 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-22 11:30:29,608 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-22 11:30:29,609 INFO [make_plots] preparing render list for 2024-07-22 WaveWatch3 forecast publish figures
2024-07-22 11:30:29,610 INFO [make_plots] added Halibut Bank figure to 2024-07-22 WaveWatch3 forecast publish render list
2024-07-22 11:30:29,612 INFO [make_plots] added Sentry Shoal figure to 2024-07-22 WaveWatch3 forecast publish render list
2024-07-22 11:30:29,612 INFO [make_plots] starting to render wwatch3 forecast publish 22jul24 figures
2024-07-22 11:30:30,169 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-22 11:30:35,801 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-22 11:30:47,394 INFO [make_plots] finished rendering wwatch3 forecast publish 22jul24 figures
2024-07-22 11:30:47,395 INFO [make_plots] wwatch3 publish plots for 2024-07-22 forecast completed
2024-07-22 11:30:47,398 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-22 12:00:28,583 INFO [download_results] nowcast-green 2024-07-22 results files from arbutus.cloud-nowcast downloaded
2024-07-22 12:00:28,591 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 12:00:28,777 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 12:00:28,783 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-07-22 12:00:28,788 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 12:00:28,793 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 12:00:28,798 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-07-22'], host='localhost')
2024-07-22 12:00:30,574 INFO [ping_erddap] running in process 731052
2024-07-22 12:00:30,576 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 12:00:30,577 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-22 12:00:30,578 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-22 12:00:30,592 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-07-22 12:00:30,595 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-22 12:00:32,166 INFO [make_averaged_dataset] running in process 731053
2024-07-22 12:00:32,168 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 12:00:32,169 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2024-07-22 12:00:32,170 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-22 12:00:32,171 INFO [make_averaged_dataset] creating day-averaged dataset for 22-Jul-2024 biology
2024-07-22 12:00:32,173 INFO [make_averaged_dataset] running in process 731054
2024-07-22 12:00:32,175 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 12:00:32,176 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2024-07-22 12:00:32,176 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-22 12:00:32,178 INFO [make_averaged_dataset] creating day-averaged dataset for 22-Jul-2024 chemistry
2024-07-22 12:00:32,221 INFO [make_averaged_dataset] running in process 731055
2024-07-22 12:00:32,222 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 12:00:32,223 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2024-07-22 12:00:32,225 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-22 12:00:32,226 INFO [make_averaged_dataset] creating day-averaged dataset for 22-Jul-2024 physics
2024-07-22 12:00:33,655 INFO [make_plots] running in process 731050
2024-07-22 12:00:33,656 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 12:00:33,657 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-22 12:00:33,658 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-22 12:00:34,063 INFO [make_plots] preparing render list for 2024-07-22 NEMO nowcast-green research figures
2024-07-22 12:00:35,863 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:36,604 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:37,353 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:38,077 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:38,803 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:39,557 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:40,297 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:41,135 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:41,978 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:42,813 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:43,653 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:44,395 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:45,165 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:45,167 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:45,169 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:45,170 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-07-22 NEMO nowcast-green research render list
2024-07-22 12:00:45,172 INFO [make_plots] starting to render nemo nowcast-green research 22jul24 figures
2024-07-22 12:01:15,687 INFO [make_averaged_dataset] day-averaged dataset for 22-Jul-2024 physics created
2024-07-22 12:01:15,691 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-22 12:02:20,338 INFO [make_averaged_dataset] day-averaged dataset for 22-Jul-2024 chemistry created
2024-07-22 12:02:20,342 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-22 12:02:36,812 INFO [make_averaged_dataset] day-averaged dataset for 22-Jul-2024 biology created
2024-07-22 12:02:36,815 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-22 13:04:43,257 INFO [make_plots] finished rendering nemo nowcast-green research 22jul24 figures
2024-07-22 13:04:43,269 INFO [make_plots] nemo research plots for 2024-07-22 nowcast-green completed
2024-07-22 13:04:43,328 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-22 13:34:45,134 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-22 13:35:11,276 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-22 15:03:12,072 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240722/18/
2024-07-22 15:03:12,074 INFO [collect_weather] 2024-07-22 2.5km weather forecast 18 collection complete
2024-07-22 15:03:12,077 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-22 15:03:12,427 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-07-22 15:03:12,433 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-07-22 15:03:12,438 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-07-22 15:03:12,443 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-07-23'], host='localhost')
2024-07-22 15:03:12,836 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240722/18/
2024-07-22 15:03:12,837 INFO [crop_gribs] 2024-07-22 18 GRIBs cropping complete
2024-07-22 15:03:12,838 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-22 15:03:13,663 INFO [download_weather] running in process 732422
2024-07-22 15:03:13,664 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 15:03:13,665 INFO [download_weather] publishing log messages to tcp://*:5562
2024-07-22 15:03:13,666 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-22 15:03:13,667 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240722
2024-07-22 15:03:13,668 INFO [download_weather] running in process 732424
2024-07-22 15:03:13,669 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 15:03:13,670 INFO [download_weather] publishing log messages to tcp://*:5565
2024-07-22 15:03:13,671 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-22 15:03:13,672 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240722
2024-07-22 15:03:13,687 INFO [collect_weather] running in process 732425
2024-07-22 15:03:13,688 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 15:03:13,690 INFO [collect_weather] publishing log messages to tcp://*:5566
2024-07-22 15:03:13,691 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-22 15:03:13,705 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-07-22 15:03:14,722 INFO [crop_gribs] running in process 732426
2024-07-22 15:03:14,723 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 15:03:14,724 INFO [crop_gribs] publishing log messages to tcp://*:5567
2024-07-22 15:03:14,725 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-22 15:03:14,726 INFO [crop_gribs] cropping 2024-07-23 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-07-22 15:03:14,739 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240723/00/
2024-07-22 15:05:44,556 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-22 15:05:44,918 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-07-01'], host='localhost')
2024-07-22 15:05:44,925 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 15:05:44,930 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 15:05:46,186 INFO [download_results] running in process 732474
2024-07-22 15:05:46,188 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 15:05:46,189 INFO [download_results] publishing log messages to tcp://*:5569
2024-07-22 15:05:46,190 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 15:05:46,191 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jul18 to /ocean/sallen/timbucktwo/oxygen
2024-07-22 15:05:50,255 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-22 15:06:35,495 INFO [download_results] hindcast 2018-07-01 results files from optimum-hindcast downloaded
2024-07-22 15:06:35,506 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 15:06:35,707 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-07-01'], host='localhost')
2024-07-22 15:06:36,834 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-22 15:08:27,618 INFO [download_weather] 2024-07-22 1km weather forecast 00 downloads complete
2024-07-22 15:08:27,622 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-22 15:08:33,175 INFO [download_weather] 2024-07-22 1km weather forecast 12 downloads complete
2024-07-22 15:08:33,179 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-22 16:30:56,183 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-22 16:30:56,589 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-07-06'], host='localhost')
2024-07-22 16:30:56,596 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 16:30:56,602 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 16:30:57,733 INFO [download_results] running in process 732981
2024-07-22 16:30:57,734 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 16:30:57,735 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-22 16:30:57,736 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 16:30:57,737 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06jul18 to /ocean/sallen/timbucktwo/oxygen
2024-07-22 16:31:02,007 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-22 16:31:38,089 INFO [download_results] hindcast 2018-07-06 results files from optimum-hindcast downloaded
2024-07-22 16:31:38,099 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 16:31:38,295 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-07-06'], host='localhost')
2024-07-22 16:31:39,301 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-22 18:01:06,833 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-22 18:01:07,179 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-07-11'], host='localhost')
2024-07-22 18:01:07,187 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 18:01:07,193 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 18:01:08,445 INFO [download_results] running in process 734257
2024-07-22 18:01:08,447 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 18:01:08,448 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-22 18:01:08,449 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 18:01:08,451 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11jul18 to /ocean/sallen/timbucktwo/oxygen
2024-07-22 18:01:12,550 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-22 18:01:48,610 INFO [download_results] hindcast 2018-07-11 results files from optimum-hindcast downloaded
2024-07-22 18:01:48,619 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 18:01:48,783 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-07-11'], host='localhost')
2024-07-22 18:01:50,030 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-22 19:26:18,160 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-22 19:26:18,496 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-07-16'], host='localhost')
2024-07-22 19:26:18,504 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 19:26:18,509 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 19:26:19,758 INFO [download_results] running in process 734554
2024-07-22 19:26:19,760 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 19:26:19,761 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-22 19:26:19,762 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 19:26:19,763 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16jul18 to /ocean/sallen/timbucktwo/oxygen
2024-07-22 19:26:23,857 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-22 19:26:59,513 INFO [download_results] hindcast 2018-07-16 results files from optimum-hindcast downloaded
2024-07-22 19:26:59,524 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 19:26:59,724 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-07-16'], host='localhost')
2024-07-22 19:27:01,010 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-22 20:51:28,500 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-22 20:51:28,832 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-07-21'], host='localhost')
2024-07-22 20:51:28,838 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 20:51:28,843 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 20:51:30,025 INFO [download_results] running in process 734942
2024-07-22 20:51:30,026 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 20:51:30,027 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-22 20:51:30,028 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 20:51:30,029 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21jul18 to /ocean/sallen/timbucktwo/oxygen
2024-07-22 20:51:34,196 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-22 20:52:09,997 INFO [download_results] hindcast 2018-07-21 results files from optimum-hindcast downloaded
2024-07-22 20:52:10,005 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 20:52:10,204 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-07-21'], host='localhost')
2024-07-22 20:52:11,372 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-22 21:06:43,473 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240723/00/
2024-07-22 21:06:43,474 INFO [collect_weather] 2024-07-23 2.5km weather forecast 00 collection complete
2024-07-22 21:06:43,477 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-22 21:06:43,821 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-07-22 21:06:43,827 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-07-23'], host='localhost')
2024-07-22 21:06:43,868 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240723/00/
2024-07-22 21:06:43,869 INFO [crop_gribs] 2024-07-23 00 GRIBs cropping complete
2024-07-22 21:06:43,872 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-22 21:06:45,030 INFO [collect_weather] running in process 735031
2024-07-22 21:06:45,031 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 21:06:45,032 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-22 21:06:45,033 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-22 21:06:45,048 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-07-22 21:06:45,801 INFO [crop_gribs] running in process 735033
2024-07-22 21:06:45,802 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 21:06:45,803 INFO [crop_gribs] publishing log messages to tcp://*:5564
2024-07-22 21:06:45,805 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-22 21:06:45,806 INFO [crop_gribs] cropping 2024-07-23 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-07-22 21:06:45,820 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240723/06/
2024-07-22 22:36:40,645 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-22 22:36:40,992 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-07-26'], host='localhost')
2024-07-22 22:36:41,000 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 22:36:41,004 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-22 22:36:42,240 INFO [download_results] running in process 735421
2024-07-22 22:36:42,241 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 22:36:42,242 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-22 22:36:42,244 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-22 22:36:42,245 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jul18 to /ocean/sallen/timbucktwo/oxygen
2024-07-22 22:36:46,473 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-22 22:37:27,287 INFO [download_results] hindcast 2018-07-26 results files from optimum-hindcast downloaded
2024-07-22 22:37:27,298 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-22 22:37:27,498 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-07-26'], host='localhost')
2024-07-22 22:37:28,724 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-22 22:37:28,929 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-jul', 'robot.graham'], host='localhost')
2024-07-22 22:37:30,171 INFO [archive_tarball] running in process 735469
2024-07-22 22:37:30,172 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-22 22:37:30,173 INFO [archive_tarball] publishing log messages to tcp://*:5565
2024-07-22 22:37:30,175 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-22 22:37:30,176 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jul18.tar from /ocean/sallen/timbucktwo/oxygen/*jul18/
2024-07-22 22:43:05,761 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jul18.index from /ocean/dlatorne/oxygen-jul18.tar
2024-07-22 22:43:05,843 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jul18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-22 22:51:05,291 INFO [archive_tarball] hindcast *jul18 results files archived to robot.graham
2024-07-22 22:51:05,295 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-23 00:01:50,524 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 00:01:50,874 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-08-01'], host='localhost')
2024-07-23 00:01:50,881 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 00:01:50,886 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 00:01:52,288 INFO [download_results] running in process 735889
2024-07-23 00:01:52,289 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 00:01:52,290 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-23 00:01:52,291 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 00:01:52,292 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01aug18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 00:01:56,476 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 00:02:31,948 INFO [download_results] hindcast 2018-08-01 results files from optimum-hindcast downloaded
2024-07-23 00:02:31,958 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 00:02:32,159 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-08-01'], host='localhost')
2024-07-23 00:02:33,313 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 01:32:01,196 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 01:32:01,541 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-08-06'], host='localhost')
2024-07-23 01:32:01,550 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 01:32:01,555 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 01:32:02,817 INFO [download_results] running in process 736435
2024-07-23 01:32:02,818 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 01:32:02,820 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-23 01:32:02,821 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 01:32:02,822 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06aug18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 01:32:07,098 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 01:32:42,793 INFO [download_results] hindcast 2018-08-06 results files from optimum-hindcast downloaded
2024-07-23 01:32:42,804 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 01:32:43,005 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-08-06'], host='localhost')
2024-07-23 01:32:44,366 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 02:57:11,410 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-23 02:57:11,780 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-08-11'], host='localhost')
2024-07-23 02:57:11,787 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 02:57:11,793 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-23 02:57:13,064 INFO [download_results] running in process 737199
2024-07-23 02:57:13,065 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 02:57:13,066 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-23 02:57:13,067 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 02:57:13,068 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11aug18 to /ocean/sallen/timbucktwo/oxygen
2024-07-23 02:57:17,299 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-23 02:57:53,657 INFO [download_results] hindcast 2018-08-11 results files from optimum-hindcast downloaded
2024-07-23 02:57:53,667 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 02:57:53,865 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-08-11'], host='localhost')
2024-07-23 02:57:55,209 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-23 03:03:15,765 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240723/06/
2024-07-23 03:03:15,766 INFO [collect_weather] 2024-07-23 2.5km weather forecast 06 collection complete
2024-07-23 03:03:15,769 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-23 03:03:16,109 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,115 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,118 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,123 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,128 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,134 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,137 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,145 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,149 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,154 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,162 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,168 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,173 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,178 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,188 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-07-23 03:03:16,195 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-07-23 03:03:16,207 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-07-23 03:03:16,216 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-07-22'], host='localhost')
2024-07-23 03:03:16,239 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-07-23 03:03:16,254 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-07-23 03:03:16,273 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-07-23 03:03:17,148 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240723/06/
2024-07-23 03:03:17,149 INFO [crop_gribs] 2024-07-23 06 GRIBs cropping complete
2024-07-23 03:03:17,151 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-23 03:03:17,978 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-07-23 03:03:19,185 INFO [collect_NeahBay_ssh] running in process 737281
2024-07-23 03:03:19,186 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:19,187 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2024-07-23 03:03:19,188 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:19,189 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-07-23
2024-07-23 03:03:19,218 INFO [collect_weather] running in process 737282
2024-07-23 03:03:19,219 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:19,220 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-07-23 03:03:19,222 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:19,245 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-07-23 03:03:20,103 INFO [collect_river_data] running in process 737264
2024-07-23 03:03:20,104 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,106 INFO [collect_river_data] publishing log messages to tcp://*:5565
2024-07-23 03:03:20,107 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,108 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-07-22
2024-07-23 03:03:20,186 INFO [collect_river_data] running in process 737266
2024-07-23 03:03:20,187 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,189 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-07-23 03:03:20,190 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,191 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-07-22
2024-07-23 03:03:20,199 INFO [collect_river_data] running in process 737268
2024-07-23 03:03:20,200 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,202 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-23 03:03:20,204 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,205 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-07-22
2024-07-23 03:03:20,207 INFO [collect_river_data] running in process 737263
2024-07-23 03:03:20,209 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,210 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-23 03:03:20,210 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,211 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-07-22
2024-07-23 03:03:20,256 INFO [collect_river_data] running in process 737267
2024-07-23 03:03:20,257 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,258 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-23 03:03:20,259 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,259 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-07-22
2024-07-23 03:03:20,260 INFO [collect_river_data] running in process 737272
2024-07-23 03:03:20,261 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,263 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-23 03:03:20,263 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,264 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-07-22
2024-07-23 03:03:20,320 INFO [collect_river_data] running in process 737273
2024-07-23 03:03:20,321 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,324 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-07-23 03:03:20,326 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,328 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-07-22
2024-07-23 03:03:20,329 INFO [collect_river_data] running in process 737274
2024-07-23 03:03:20,330 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,334 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-07-23 03:03:20,334 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,335 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-07-22
2024-07-23 03:03:20,363 INFO [collect_river_data] running in process 737269
2024-07-23 03:03:20,364 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,365 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-07-23 03:03:20,367 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,368 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-07-22
2024-07-23 03:03:20,379 INFO [collect_river_data] running in process 737265
2024-07-23 03:03:20,380 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,381 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-07-23 03:03:20,383 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,384 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-07-22
2024-07-23 03:03:20,389 INFO [collect_river_data] running in process 737275
2024-07-23 03:03:20,390 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,391 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-07-23 03:03:20,392 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,393 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-07-22
2024-07-23 03:03:20,404 INFO [collect_river_data] running in process 737270
2024-07-23 03:03:20,406 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,407 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-07-23 03:03:20,408 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,409 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-07-22
2024-07-23 03:03:20,410 INFO [collect_river_data] running in process 737261
2024-07-23 03:03:20,411 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,412 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-07-23 03:03:20,412 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,413 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-07-22
2024-07-23 03:03:20,414 INFO [collect_river_data] running in process 737276
2024-07-23 03:03:20,414 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,415 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-07-23 03:03:20,416 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,416 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-07-22
2024-07-23 03:03:20,699 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-07-23 03:03:20,700 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-07-22 completed
2024-07-23 03:03:20,702 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:20,823 INFO [collect_NeahBay_ssh] 2024-07-23T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-07-23 03:03:20,827 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-07-22 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-07-23 03:03:20,828 INFO [collect_river_data] ECCC Fraser river data collection for 2024-07-22 completed
2024-07-23 03:03:20,890 INFO [get_vfpa_hadcp] running in process 737280
2024-07-23 03:03:20,891 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:20,892 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5590
2024-07-23 03:03:20,892 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:20,893 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-22
2024-07-23 03:03:20,956 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-07-23 03:03:20,957 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-07-22 completed
2024-07-23 03:03:21,016 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-07-23 03:03:21,017 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-07-22 completed
2024-07-23 03:03:21,080 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-07-23 03:03:21,080 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-07-22 completed
2024-07-23 03:03:21,084 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-07-23 03:03:21,085 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-07-22 completed
2024-07-23 03:03:21,113 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-07-23 03:03:21,114 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-07-22 completed
2024-07-23 03:03:21,134 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-07-22 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-07-23 03:03:21,135 INFO [collect_river_data] ECCC Capilano river data collection for 2024-07-22 completed
2024-07-23 03:03:21,140 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-07-23 03:03:21,141 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-07-22 completed
2024-07-23 03:03:21,144 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-07-23 03:03:21,145 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-07-22 completed
2024-07-23 03:03:21,150 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-07-23 03:03:21,151 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-07-22 completed
2024-07-23 03:03:21,160 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-23 03:03:21,197 INFO [crop_gribs] running in process 737283
2024-07-23 03:03:21,198 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:21,199 INFO [crop_gribs] publishing log messages to tcp://*:5591
2024-07-23 03:03:21,199 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:21,200 INFO [crop_gribs] cropping 2024-07-23 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-07-23 03:03:21,202 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-07-23 03:03:21,202 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-07-22 completed
2024-07-23 03:03:21,215 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240723/12/
2024-07-23 03:03:21,237 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-07-22 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-07-23 03:03:21,237 INFO [collect_river_data] ECCC Englishman river data collection for 2024-07-22 completed
2024-07-23 03:03:21,242 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-07-22 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-07-23 03:03:21,243 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-07-22 completed
2024-07-23 03:03:21,398 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 03:03:21,403 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:21,529 INFO [get_onc_ferry] running in process 737279
2024-07-23 03:03:21,530 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:21,531 INFO [get_onc_ferry] publishing log messages to tcp://*:5565
2024-07-23 03:03:21,532 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:21,532 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-07-22
2024-07-23 03:03:21,548 INFO [get_onc_ctd] running in process 737278
2024-07-23 03:03:21,549 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:21,550 INFO [get_onc_ctd] publishing log messages to tcp://*:5592
2024-07-23 03:03:21,551 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:21,552 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-07-22
2024-07-23 03:03:21,637 INFO [get_onc_ctd] running in process 737277
2024-07-23 03:03:21,638 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:21,639 INFO [get_onc_ctd] publishing log messages to tcp://*:5593
2024-07-23 03:03:21,640 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:21,641 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-07-22
2024-07-23 03:03:21,659 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:21,759 INFO [grib_to_netcdf] running in process 737528
2024-07-23 03:03:21,760 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:21,762 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2024-07-23 03:03:21,762 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:21,764 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-23 forecast2 run
2024-07-23 03:03:21,872 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:22,053 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:22,268 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:22,457 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:22,647 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:22,841 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:23,055 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:23,179 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-07-22
2024-07-23 03:03:23,194 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-22 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202407.nc
2024-07-23 03:03:23,195 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-07 netcdf file
2024-07-23 03:03:23,262 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:23,459 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:23,668 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:23,745 CRITICAL [get_onc_ctd] 2024-07-22 ONC SCVIP CTD T&S file creation failed
2024-07-23 03:03:23,987 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-23 03:03:24,106 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2024-07-23 03:03:24,319 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2024-07-23 03:03:24,399 INFO [make_ssh_files] running in process 737719
2024-07-23 03:03:24,400 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:24,402 INFO [make_ssh_files] publishing log messages to tcp://*:5566
2024-07-23 03:03:24,403 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:24,403 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-07-23 NOAA Neah Bay 00Z observation and forecast values
2024-07-23 03:03:25,613 INFO [ping_erddap] running in process 737764
2024-07-23 03:03:25,614 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:25,616 INFO [ping_erddap] publishing log messages to tcp://*:5568
2024-07-23 03:03:25,617 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:25,618 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2024-07-23 03:03:25,620 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 03:03:29,161 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc
2024-07-23 03:03:29,344 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc
2024-07-23 03:03:29,531 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d20.nc
2024-07-23 03:03:29,719 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d21.nc
2024-07-23 03:03:29,899 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d22.nc
2024-07-23 03:03:30,132 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d23.nc
2024-07-23 03:03:30,322 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d24.nc
2024-07-23 03:03:30,405 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d24.nc
2024-07-23 03:03:30,507 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d25.nc
2024-07-23 03:03:30,690 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d26.nc
2024-07-23 03:03:30,890 INFO [make_ssh_files] sea surface height boundary file for 2024-07-23 forecast2 run created
2024-07-23 03:03:30,894 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-23 03:03:31,101 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-23 03:03:31,104 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-23 03:03:32,484 INFO [make_runoff_file] running in process 737801
2024-07-23 03:03:32,485 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:32,486 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2024-07-23 03:03:32,487 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:32,608 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-23 03:03:32,712 INFO [make_runoff_file] 2024-07-23 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-23 03:03:32,715 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-23 03:03:32,818 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d25.nc
2024-07-23 03:03:32,824 INFO [grib_to_netcdf] 2024-07-23 NEMO-atmos forcing file for forecast2 created
2024-07-23 03:03:32,923 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-23 03:03:33,011 INFO [make_v202111_runoff_file] running in process 737799
2024-07-23 03:03:33,012 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:33,013 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5568
2024-07-23 03:03:33,014 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:33,015 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-22
2024-07-23 03:03:34,103 ERROR [make_v202111_runoff_file] no 2024-07-22 discharge obs for Skagit_MountVernon: patching
2024-07-23 03:03:34,595 ERROR [make_v202111_runoff_file] no 2024-07-22 discharge obs for Snohomish_Monroe: patching
2024-07-23 03:03:34,800 ERROR [make_v202111_runoff_file] no 2024-07-22 discharge obs for Nisqually_McKenna: patching
2024-07-23 03:03:34,897 ERROR [make_v202111_runoff_file] no 2024-07-22 discharge obs for Greenwater_Greenwater: patching
2024-07-23 03:03:36,200 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-22: /results/forcing/rivers/R202108Dailies_y2024m07d22.nc
2024-07-23 03:03:36,201 INFO [make_v202111_runoff_file] 2024-07-22 runoff file creation completed
2024-07-23 03:03:36,204 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-23 03:03:36,406 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-07-23 03:03:36,410 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-07-23 03:03:36,415 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-07-23 03:03:36,419 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-07-23 03:03:37,587 INFO [upload_forcing] running in process 737840
2024-07-23 03:03:37,589 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:37,590 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-07-23 03:03:37,591 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:37,626 INFO [upload_forcing] running in process 737841
2024-07-23 03:03:37,627 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:37,628 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-23 03:03:37,629 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:37,678 INFO [upload_forcing] running in process 737842
2024-07-23 03:03:37,679 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:37,680 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-23 03:03:37,681 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:37,780 INFO [upload_forcing] running in process 737843
2024-07-23 03:03:37,782 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:37,783 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-23 03:03:37,784 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:39,218 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d22.nc
2024-07-23 03:03:39,329 INFO [upload_forcing] forecast2 2024-07-23 forcing files upload to optimum-hindcast completed
2024-07-23 03:03:39,332 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 03:03:39,334 INFO [upload_forcing] forecast2 2024-07-23 forcing files upload to orcinus-nowcast-agrif completed
2024-07-23 03:03:39,556 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 03:03:40,215 INFO [upload_forcing] forecast2 2024-07-23 forcing files upload to arbutus.cloud-nowcast completed
2024-07-23 03:03:40,219 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 03:03:40,419 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 03:03:40,578 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-07-22
2024-07-23 03:03:41,273 WARNING [get_onc_ferry] No ONC TWDP TSG temperature,conductivity,salinity data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:41,278 WARNING [get_onc_ferry] No ONC TWDP TSG temperature data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:41,280 WARNING [get_onc_ferry] No ONC TWDP TSG conductivity data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:41,281 WARNING [get_onc_ferry] No ONC TWDP TSG salinity data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:41,283 WARNING [get_onc_ctd] no SEVIP salinity data
2024-07-23 03:03:41,284 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-07-22
2024-07-23 03:03:41,533 INFO [get_onc_ctd] 2024-07-22 ONC SEVIP CTD T&S file created
2024-07-23 03:03:41,536 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-07-23 03:03:41,579 INFO [make_forcing_links] running in process 737870
2024-07-23 03:03:41,580 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:41,581 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-07-23 03:03:41,583 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:41,741 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-07-23 03:03:42,316 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation,oxygen_corrected,temperature data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:42,322 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:42,324 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:42,325 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR temperature data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:42,326 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-07-22
2024-07-23 03:03:42,829 INFO [make_forcing_links] forecast2 2024-07-23 forcing file links on arbutus.cloud-nowcast created
2024-07-23 03:03:42,833 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-23 03:03:42,917 INFO [ping_erddap] running in process 737877
2024-07-23 03:03:42,918 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:42,919 INFO [ping_erddap] publishing log messages to tcp://*:5566
2024-07-23 03:03:42,921 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:42,922 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-07-23 03:03:43,007 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence,chlorophyll,turbidity data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:43,013 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:43,015 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL chlorophyll data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:43,017 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL turbidity data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:43,018 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-07-22
2024-07-23 03:03:43,061 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 03:03:43,065 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 03:03:43,914 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:43,919 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:43,921 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2024-07-22; substituting empty dataset
2024-07-23 03:03:43,923 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-07-22
2024-07-23 03:03:45,173 INFO [run_NEMO] running in process 27251
2024-07-23 03:03:45,175 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:45,176 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-23 03:03:48,682 INFO [upload_forcing] forecast2 2024-07-23 forcing files upload to robot.graham completed
2024-07-23 03:03:48,686 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-23 03:03:52,992 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:54,032 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/22jul24forecast2_2024-07-23T030353.076352-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-23 03:03:54,058 INFO [run_NEMO] forecast2 NEMO run for 2024-07-23 on arbutus.cloud-nowcast started
2024-07-23 03:03:54,062 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-23 03:03:54,264 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-23 03:03:56,239 INFO [watch_NEMO] running in process 27505
2024-07-23 03:03:56,241 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:03:56,242 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-23 03:03:56,243 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:03:56,314 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-23 03:04:01,314 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-07-22
2024-07-23 03:04:09,641 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-07-22
2024-07-23 03:04:17,526 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-07-22; substituting empty dataset
2024-07-23 03:04:17,526 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-07-22
2024-07-23 03:04:25,443 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-22; substituting empty dataset
2024-07-23 03:04:25,445 WARNING [get_onc_ferry] ONC TWDP TSG temperature data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,466 WARNING [get_onc_ferry] ONC TWDP TSG conductivity data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,528 WARNING [get_onc_ferry] ONC TWDP TSG salinity data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,546 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,563 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,580 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR temperature data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,598 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,615 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL chlorophyll data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,632 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL turbidity data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,649 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,667 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,769 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:04:25,786 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-22 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-23 03:05:45,409 INFO [get_onc_ferry] 2024-07-22 ONC TWDP ferry data file created
2024-07-23 03:05:45,411 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-07-23 03:05:45,616 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-07-23 03:05:46,867 INFO [ping_erddap] running in process 737894
2024-07-23 03:05:46,869 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:05:46,870 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-23 03:05:46,871 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:05:46,873 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-07-23 03:05:46,875 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 03:08:56,316 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17805338 = 2024-07-24 05:04:40 UTC, 16.9% complete
2024-07-23 03:13:56,324 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17805818 = 2024-07-24 10:24:40 UTC, 34.7% complete
2024-07-23 03:18:56,339 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17806329 = 2024-07-24 16:05:20 UTC, 53.6% complete
2024-07-23 03:23:56,321 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17806823 = 2024-07-24 21:34:40 UTC, 72.0% complete
2024-07-23 03:29:03,230 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17807293 = 2024-07-25 02:48:00 UTC, 89.4% complete
2024-07-23 03:33:56,327 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-07-23 03:33:56,331 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-23 03:33:56,691 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 03:33:56,699 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 03:33:56,703 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-23 03:33:57,994 INFO [download_results] running in process 737962
2024-07-23 03:33:57,995 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:33:57,996 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-23 03:33:57,997 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:33:57,998 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/22jul24 to /results/SalishSea/forecast2.202111
2024-07-23 03:34:00,182 INFO [make_ww3_wind_file] running in process 27773
2024-07-23 03:34:00,183 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:34:00,184 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-23 03:34:00,186 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:34:00,187 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-23 forecast2 run
2024-07-23 03:34:00,257 INFO [make_ww3_current_file] running in process 27738
2024-07-23 03:34:00,258 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:34:00,259 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-23 03:34:00,260 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:34:00,261 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-23 forecast2 run
2024-07-23 03:34:09,003 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-23 forecast2 run
2024-07-23 03:34:09,006 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-23 03:36:07,380 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-23 forecast2 run
2024-07-23 03:36:07,382 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-23 03:36:07,590 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-23'], host='arbutus.cloud-nowcast')
2024-07-23 03:36:18,692 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-23 03:36:18,899 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-23 03:36:30,747 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-23 03:41:30,292 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 72, 13.3% complete
2024-07-23 03:46:30,783 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 147, 27.2% complete
2024-07-23 03:51:30,756 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 228, 42.2% complete
2024-07-23 03:56:30,744 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 309, 57.2% complete
2024-07-23 03:58:53,178 INFO [download_results] forecast2 2024-07-22 results files from arbutus.cloud-nowcast downloaded
2024-07-23 03:58:53,182 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-23 03:58:53,387 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-23 03:58:55,713 INFO [make_CHS_currents_file] running in process 738014
2024-07-23 03:58:55,714 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:58:55,715 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-07-23 03:58:55,716 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:59:30,439 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-22 for forecast2
2024-07-23 03:59:30,442 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-23 03:59:30,632 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-23 03:59:31,801 INFO [update_forecast_datasets] running in process 738042
2024-07-23 03:59:31,802 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 03:59:31,804 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-07-23 03:59:31,805 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-23 03:59:31,806 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/22jul24 run
2024-07-23 04:01:30,086 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 387, 71.7% complete
2024-07-23 04:03:44,438 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/22jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-23 04:03:44,439 INFO [update_forecast_datasets] nemo 2024-07-22 forecast2 rolling forecast datasets updated
2024-07-23 04:03:44,441 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-23 04:03:44,634 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-23 04:03:44,638 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-07-22'], host='localhost')
2024-07-23 04:03:44,642 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-23 04:03:45,876 INFO [ping_erddap] running in process 738118
2024-07-23 04:03:45,877 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:03:45,878 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-23 04:03:45,879 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:03:45,887 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-23 04:03:45,889 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 04:03:47,001 INFO [make_surface_current_tiles] running in process 738121
2024-07-23 04:03:47,002 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:03:47,004 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2024-07-23 04:03:47,006 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:03:47,027 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-22 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/22jul24
2024-07-23 04:03:49,793 INFO [make_plots] running in process 738120
2024-07-23 04:03:49,797 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:03:49,798 INFO [make_plots] publishing log messages to tcp://*:5565
2024-07-23 04:03:49,799 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:03:50,507 INFO [make_plots] preparing render list for 2024-07-22 NEMO forecast2 publish figures
2024-07-23 04:03:50,630 INFO [make_plots] added Website_thumbnail figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,632 INFO [make_plots] added Threshold_website figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,633 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,634 INFO [make_plots] added Boundary Bay figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,635 INFO [make_plots] added Campbell River figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,636 INFO [make_plots] added Cherry Point figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,637 INFO [make_plots] added Friday Harbor figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,638 INFO [make_plots] added Halfmoon Bay figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,638 INFO [make_plots] added Nanaimo figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,639 INFO [make_plots] added Neah Bay figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,640 INFO [make_plots] added New Westminster figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,640 INFO [make_plots] added Patricia Bay figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,641 INFO [make_plots] added Point Atkinson figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,642 INFO [make_plots] added Port Renfrew figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,642 INFO [make_plots] added Sand Heads figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,643 INFO [make_plots] added Sandy Cove figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,644 INFO [make_plots] added Squamish figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,645 INFO [make_plots] added Victoria figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,645 INFO [make_plots] added Woodwards Landing figure to 2024-07-22 NEMO forecast2 publish render list
2024-07-23 04:03:50,646 INFO [make_plots] starting to render nemo forecast2 publish 22jul24 figures
2024-07-23 04:06:30,085 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 468, 86.7% complete
2024-07-23 04:11:30,089 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-23 04:14:59,060 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/22jul24
2024-07-23 04:16:30,093 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-23 04:16:30,103 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-23 04:16:30,481 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 04:16:31,689 INFO [download_wwatch3_results] running in process 740066
2024-07-23 04:16:31,690 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:16:31,691 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566
2024-07-23 04:16:31,692 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:17:07,646 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-23 04:17:07,647 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-23 04:17:43,882 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:18:50,754 INFO [download_wwatch3_results] forecast2 2024-07-23 results files from arbutus.cloud-nowcast downloaded
2024-07-23 04:18:50,758 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-23 04:18:50,978 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 04:18:52,258 INFO [update_forecast_datasets] running in process 740094
2024-07-23 04:18:52,259 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:18:52,260 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566
2024-07-23 04:18:52,261 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:18:52,263 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/23jul24 run
2024-07-23 04:18:52,288 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/23jul24 run
2024-07-23 04:19:13,487 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/22jul24
2024-07-23 04:19:13,498 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-22 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/22jul24
2024-07-23 04:19:13,499 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-22 forecast2 completed
2024-07-23 04:19:13,796 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-23 04:19:16,960 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/23jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-23 04:19:16,961 INFO [update_forecast_datasets] wwatch3 2024-07-23 forecast2 rolling forecast datasets updated
2024-07-23 04:19:16,964 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-23 04:19:17,293 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-23 04:19:18,471 INFO [ping_erddap] running in process 740108
2024-07-23 04:19:18,472 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:19:18,473 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-23 04:19:18,474 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:19:18,476 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-23 04:19:18,478 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-23 04:19:18,758 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-07-23'], host='localhost')
2024-07-23 04:19:22,228 INFO [make_plots] running in process 740113
2024-07-23 04:19:22,231 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:19:22,232 INFO [make_plots] publishing log messages to tcp://*:5562
2024-07-23 04:19:22,233 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:19:22,234 INFO [make_plots] preparing render list for 2024-07-23 WaveWatch3 forecast2 publish figures
2024-07-23 04:19:22,235 INFO [make_plots] added Halibut Bank figure to 2024-07-23 WaveWatch3 forecast2 publish render list
2024-07-23 04:19:22,236 INFO [make_plots] added Sentry Shoal figure to 2024-07-23 WaveWatch3 forecast2 publish render list
2024-07-23 04:19:22,237 INFO [make_plots] starting to render wwatch3 forecast2 publish 23jul24 figures
2024-07-23 04:19:22,706 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-23 04:19:28,243 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-23 04:19:34,033 INFO [make_plots] finished rendering wwatch3 forecast2 publish 23jul24 figures
2024-07-23 04:19:34,034 INFO [make_plots] wwatch3 publish plots for 2024-07-23 forecast2 completed
2024-07-23 04:19:34,037 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-23 04:19:35,122 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:19:35,488 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:20:10,194 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:21:18,106 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:21:53,237 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:22:28,377 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:23:03,727 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:23:37,232 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:24:12,268 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:24:47,609 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:24:47,961 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:25:22,574 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:25:56,506 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-18 00:00:00Z to 2024-07-25 05:55:00Z
2024-07-23 04:26:31,118 INFO [make_plots] finished rendering nemo forecast2 publish 22jul24 figures
2024-07-23 04:26:31,121 INFO [make_plots] nemo publish plots for 2024-07-22 forecast2 completed
2024-07-23 04:26:31,129 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-23 04:26:31,454 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-07-22'], host='localhost')
2024-07-23 04:26:34,447 INFO [make_feeds] running in process 740185
2024-07-23 04:26:34,450 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:26:34,451 INFO [make_feeds] publishing log messages to tcp://*:5562
2024-07-23 04:26:34,453 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:27:02,156 INFO [make_feeds] ATOM feeds for 2024-07-22 forecast2 run completed
2024-07-23 04:27:02,160 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-23 04:27:02,468 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-07-23 04:27:03,583 INFO [clear_checklist] running in process 740208
2024-07-23 04:27:03,584 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:27:03,585 INFO [clear_checklist] publishing log messages to tcp://*:5562
2024-07-23 04:27:03,586 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:27:03,587 INFO [clear_checklist] requesting that manager clear system state checklist
2024-07-23 04:27:03,589 INFO [manager] writing checklist to log file
2024-07-23 04:27:03,638 INFO [manager] checklist cleared
2024-07-23 04:27:03,640 INFO [clear_checklist] nowcast system checklist cleared
2024-07-23 04:27:03,643 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-07-23 04:27:04,618 INFO [rotate_logs] running in process 740213
2024-07-23 04:27:04,619 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-23 04:27:04,620 INFO [rotate_logs] publishing log messages to tcp://*:5562
2024-07-23 04:27:04,621 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-07-23 04:27:04,622 INFO [rotate_logs] rotating log files