2024-07-19 04:32:12,485 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log 2024-07-19 04:32:12,490 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log 2024-07-19 04:32:12,494 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log 2024-07-19 04:32:12,495 INFO [rotate_logs] log files rotated 2024-07-19 04:32:12,500 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker 2024-07-19 04:32:42,847 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-dec18.index from /ocean/dlatorne/oxygen-dec18.tar 2024-07-19 04:32:42,927 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-dec18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-07-19 04:50:46,985 INFO [archive_tarball] hindcast *dec18 results files archived to robot.graham 2024-07-19 04:50:46,989 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-07-19 05:51:56,805 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 05:51:56,953 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-01-01'], host='localhost') 2024-07-19 05:51:56,959 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 05:51:56,964 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 05:51:58,162 INFO [download_results] running in process 683525 2024-07-19 05:51:58,164 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 05:51:58,165 INFO [download_results] publishing log messages to tcp://*:5565 2024-07-19 05:51:58,166 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 05:51:58,167 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jan19 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 05:52:02,243 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 05:52:37,937 INFO [download_results] hindcast 2019-01-01 results files from optimum-hindcast downloaded 2024-07-19 05:52:37,948 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 05:52:37,955 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-01-01'], host='localhost') 2024-07-19 05:52:38,994 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 07:56:35,312 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 07:56:54,802 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 09:02:54,121 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240719/12/ 2024-07-19 09:02:54,122 INFO [collect_weather] 2024-07-19 2.5km weather forecast 12 collection complete 2024-07-19 09:02:54,125 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-07-19 09:02:54,285 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-07-18'], host='localhost') 2024-07-19 09:02:54,292 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-07-18'], host='localhost') 2024-07-19 09:02:54,295 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-07-18'], host='localhost') 2024-07-19 09:02:54,299 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-07-18'], host='localhost') 2024-07-19 09:02:54,303 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost') 2024-07-19 09:02:54,308 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost') 2024-07-19 09:02:54,313 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost') 2024-07-19 09:02:54,317 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost') 2024-07-19 09:02:54,326 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost') 2024-07-19 09:02:54,693 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240719/12/ 2024-07-19 09:02:54,694 INFO [crop_gribs] 2024-07-19 12 GRIBs cropping complete 2024-07-19 09:02:54,695 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-07-19 09:02:54,858 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost') 2024-07-19 09:02:55,930 INFO [collect_NeahBay_ssh] running in process 687251 2024-07-19 09:02:55,931 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:55,932 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564 2024-07-19 09:02:55,933 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:55,934 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-07-19 2024-07-19 09:02:55,975 INFO [collect_weather] running in process 687254 2024-07-19 09:02:55,976 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:55,977 INFO [collect_weather] publishing log messages to tcp://*:5565 2024-07-19 09:02:55,978 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:55,994 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/ 2024-07-19 09:02:56,125 INFO [download_live_ocean] running in process 687252 2024-07-19 09:02:56,126 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:56,127 INFO [download_live_ocean] publishing log messages to tcp://*:5566 2024-07-19 09:02:56,127 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:56,128 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-07-19 2024-07-19 09:02:56,470 INFO [collect_river_data] running in process 687248 2024-07-19 09:02:56,471 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:56,472 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-07-19 09:02:56,473 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:56,474 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-07-18 2024-07-19 09:02:56,512 INFO [collect_river_data] running in process 687249 2024-07-19 09:02:56,514 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:56,515 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-07-19 09:02:56,515 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:56,516 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-07-18 2024-07-19 09:02:56,545 INFO [collect_river_data] running in process 687245 2024-07-19 09:02:56,545 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:56,546 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-07-19 09:02:56,547 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:56,547 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-07-18 2024-07-19 09:02:56,566 INFO [collect_river_data] running in process 687247 2024-07-19 09:02:56,567 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:56,568 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-07-19 09:02:56,569 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:56,569 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-07-18 2024-07-19 09:02:56,664 INFO [make_turbidity_file] running in process 687250 2024-07-19 09:02:56,665 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:56,666 INFO [make_turbidity_file] publishing log messages to tcp://*:5571 2024-07-19 09:02:56,666 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:56,667 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-07-19 2024-07-19 09:02:56,719 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12200500&startDT=2024-07-18&endDT=2024-07-18 "HTTP/1.1 200 OK" 2024-07-19 09:02:56,724 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow 2024-07-19 09:02:56,725 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-07-18 completed 2024-07-19 09:02:56,728 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-19 09:02:56,817 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12089500&startDT=2024-07-18&endDT=2024-07-18 "HTTP/1.1 200 OK" 2024-07-19 09:02:56,829 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12097500&startDT=2024-07-18&endDT=2024-07-18 "HTTP/1.1 200 OK" 2024-07-19 09:02:56,865 INFO [crop_gribs] running in process 687255 2024-07-19 09:02:56,866 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:56,868 INFO [crop_gribs] publishing log messages to tcp://*:5572 2024-07-19 09:02:56,869 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:56,870 INFO [crop_gribs] cropping 2024-07-19 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain 2024-07-19 09:02:56,884 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240719/18/ 2024-07-19 09:02:56,886 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json¶meterCd=00060&sites=12150800&startDT=2024-07-18&endDT=2024-07-18 "HTTP/1.1 200 OK" 2024-07-19 09:02:57,066 INFO [grib_to_netcdf] running in process 687280 2024-07-19 09:02:57,067 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:02:57,068 INFO [grib_to_netcdf] publishing log messages to tcp://*:5569 2024-07-19 09:02:57,069 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:02:57,070 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-19 nowcast and forecast runs 2024-07-19 09:02:57,111 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow 2024-07-19 09:02:57,112 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-07-18 completed 2024-07-19 09:02:57,114 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-19 09:02:57,119 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow 2024-07-19 09:02:57,120 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-07-18 completed 2024-07-19 09:02:57,122 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow 2024-07-19 09:02:57,123 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-07-18 completed 2024-07-19 09:02:57,126 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-19 09:02:57,133 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-19 09:02:57,947 INFO [download_live_ocean] downloaded 10148964 bytes from https://liveocean.apl.uw.edu/output/f2024.07.19/ubc.nc to /results/forcing/LiveOcean/downloaded/20240719/low_passed_UBC.nc 2024-07-19 09:02:57,948 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes 2024-07-19 09:02:58,239 INFO [collect_NeahBay_ssh] 2024-07-19T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete 2024-07-19 09:02:58,242 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-07-19 09:02:58,247 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 09:02:58,968 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240719/low_passed_UBC.nc 2024-07-19 09:02:58,970 INFO [download_live_ocean] 2024-07-19 Live Ocean file for Salish Sea western boundary downloaded 2024-07-19 09:02:58,971 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker 2024-07-19 09:02:59,126 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-07-19'], host='localhost') 2024-07-19 09:02:59,703 INFO [make_turbidity_file] 2024-07-19 Fraser River turbidity file creation complete 2024-07-19 09:02:59,704 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker 2024-07-19 09:03:00,694 INFO [make_ssh_files] running in process 687427 2024-07-19 09:03:00,696 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:00,697 INFO [make_ssh_files] publishing log messages to tcp://*:5564 2024-07-19 09:03:00,698 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:00,699 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-07-19 NOAA Neah Bay 06Z observation and forecast values 2024-07-19 09:03:01,115 INFO [make_live_ocean_files] running in process 687441 2024-07-19 09:03:01,116 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:01,118 INFO [make_live_ocean_files] publishing log messages to tcp://*:5566 2024-07-19 09:03:01,119 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:01,123 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-07-19 Live Ocean run 2024-07-19 09:03:05,208 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d15.nc 2024-07-19 09:03:05,362 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d16.nc 2024-07-19 09:03:05,530 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d17.nc 2024-07-19 09:03:05,703 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc 2024-07-19 09:03:05,892 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d19.nc 2024-07-19 09:03:06,076 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d20.nc 2024-07-19 09:03:06,254 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d21.nc 2024-07-19 09:03:06,451 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d22.nc 2024-07-19 09:03:06,564 INFO [make_ssh_files] sea surface height boundary file for 2024-07-19 nowcast run created 2024-07-19 09:03:06,568 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-07-19 09:03:06,576 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-07-19 09:03:06,581 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-07-19 09:03:06,816 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m07d19.nc 2024-07-19 09:03:07,903 INFO [make_runoff_file] running in process 687501 2024-07-19 09:03:07,904 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:07,905 INFO [make_runoff_file] publishing log messages to tcp://*:5564 2024-07-19 09:03:07,906 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:07,930 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-07-19 09:03:08,007 INFO [make_runoff_file] 2024-07-19 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-07-19 09:03:08,010 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-07-19 09:03:08,381 INFO [make_v202111_runoff_file] running in process 687499 2024-07-19 09:03:08,383 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:08,384 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564 2024-07-19 09:03:08,385 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:08,386 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-18 2024-07-19 09:03:10,955 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-18: /results/forcing/rivers/R202108Dailies_y2024m07d18.nc 2024-07-19 09:03:10,955 INFO [make_v202111_runoff_file] 2024-07-18 runoff file creation completed 2024-07-19 09:03:10,958 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-07-19 09:03:13,815 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d20.nc 2024-07-19 09:03:17,492 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d21.nc 2024-07-19 09:03:17,508 INFO [grib_to_netcdf] 2024-07-19 NEMO-atmos forcing file for nowcast+ created 2024-07-19 09:03:17,511 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-07-19 09:03:40,651 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d19.nc 2024-07-19 09:03:40,652 INFO [make_live_ocean_files] 2024-07-19 Live Ocean western boundary conditions files created 2024-07-19 09:03:40,654 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker 2024-07-19 09:03:40,665 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost') 2024-07-19 09:03:40,668 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost') 2024-07-19 09:03:40,671 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost') 2024-07-19 09:03:40,676 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost') 2024-07-19 09:03:40,680 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost') 2024-07-19 09:03:41,769 INFO [ping_erddap] running in process 687540 2024-07-19 09:03:41,770 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:41,771 INFO [ping_erddap] publishing log messages to tcp://*:5564 2024-07-19 09:03:41,772 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:41,774 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created 2024-07-19 09:03:41,776 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-19 09:03:41,832 INFO [upload_forcing] running in process 687543 2024-07-19 09:03:41,833 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:41,834 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-07-19 09:03:41,835 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:41,848 INFO [upload_forcing] running in process 687542 2024-07-19 09:03:41,850 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:41,851 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-07-19 09:03:41,852 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:41,875 INFO [upload_forcing] running in process 687545 2024-07-19 09:03:41,876 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:41,877 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-07-19 09:03:41,879 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:41,892 INFO [upload_forcing] running in process 687544 2024-07-19 09:03:41,893 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:41,894 INFO [upload_forcing] publishing log messages to tcp://*:5569 2024-07-19 09:03:41,895 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:44,136 INFO [upload_forcing] nowcast+ 2024-07-19 forcing files upload to optimum-hindcast completed 2024-07-19 09:03:44,139 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-19 09:03:44,325 INFO [upload_forcing] nowcast+ 2024-07-19 forcing files upload to orcinus-nowcast-agrif completed 2024-07-19 09:03:44,329 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-19 09:03:51,772 INFO [upload_forcing] nowcast+ 2024-07-19 forcing files upload to robot.graham completed 2024-07-19 09:03:51,777 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-19 09:03:53,597 INFO [upload_forcing] nowcast+ 2024-07-19 forcing files upload to arbutus.cloud-nowcast completed 2024-07-19 09:03:53,601 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-19 09:03:53,613 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 09:03:54,785 INFO [make_forcing_links] running in process 687579 2024-07-19 09:03:54,786 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:54,787 INFO [make_forcing_links] publishing log messages to tcp://*:5564 2024-07-19 09:03:54,788 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:56,008 INFO [make_forcing_links] nowcast+ 2024-07-19 forcing file links on arbutus.cloud-nowcast created 2024-07-19 09:03:56,012 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-07-19 09:03:56,027 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast') 2024-07-19 09:03:58,156 INFO [run_NEMO] running in process 4629 2024-07-19 09:03:58,157 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:03:58,158 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-07-19 09:03:58,168 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:03:59,059 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/19jul24nowcast_2024-07-19T090358.211543-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-07-19 09:03:59,077 INFO [run_NEMO] nowcast NEMO run for 2024-07-19 on arbutus.cloud-nowcast started 2024-07-19 09:03:59,081 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-07-19 09:03:59,096 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-07-19 09:04:01,085 INFO [watch_NEMO] running in process 4881 2024-07-19 09:04:01,086 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:04:01,087 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-07-19 09:04:01,088 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:04:01,156 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-07-19 09:09:01,157 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17794525 = 2024-07-19 04:56:00 UTC, 20.6% complete 2024-07-19 09:14:01,158 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17794987 = 2024-07-19 10:04:00 UTC, 42.0% complete 2024-07-19 09:19:01,159 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17795453 = 2024-07-19 15:14:40 UTC, 63.5% complete 2024-07-19 09:24:01,160 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17795922 = 2024-07-19 20:27:20 UTC, 85.3% complete 2024-07-19 09:27:16,737 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 09:27:16,955 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-01-01'], host='localhost') 2024-07-19 09:27:16,962 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 09:27:16,967 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 09:27:18,158 INFO [download_results] running in process 687643 2024-07-19 09:27:18,159 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:27:18,160 INFO [download_results] publishing log messages to tcp://*:5566 2024-07-19 09:27:18,161 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:27:18,162 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jan18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 09:27:22,488 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 09:27:58,489 INFO [download_results] hindcast 2018-01-01 results files from optimum-hindcast downloaded 2024-07-19 09:27:58,498 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 09:27:58,507 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-01-01'], host='localhost') 2024-07-19 09:27:59,562 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 09:29:01,162 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17796240 = 2024-07-19 23:59:20 UTC, 100.0% complete 2024-07-19 09:34:01,168 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed 2024-07-19 09:34:01,171 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-07-19 09:34:01,350 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 09:34:01,356 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 09:34:02,505 INFO [make_forcing_links] running in process 687705 2024-07-19 09:34:02,506 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:34:02,507 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2024-07-19 09:34:02,507 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:34:02,518 INFO [download_results] running in process 687706 2024-07-19 09:34:02,520 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:34:02,520 INFO [download_results] publishing log messages to tcp://*:5566 2024-07-19 09:34:02,521 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:34:02,522 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/19jul24 to /results/SalishSea/nowcast-blue.202111 2024-07-19 09:34:03,501 INFO [make_forcing_links] ssh 2024-07-19 forcing file links on arbutus.cloud-nowcast created 2024-07-19 09:34:03,505 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-07-19 09:34:03,520 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast') 2024-07-19 09:34:05,733 INFO [run_NEMO] running in process 5312 2024-07-19 09:34:05,734 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:34:05,735 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-07-19 09:34:05,736 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:34:06,732 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/19jul24forecast_2024-07-19T093405.812141-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-07-19 09:34:06,757 INFO [run_NEMO] forecast NEMO run for 2024-07-19 on arbutus.cloud-nowcast started 2024-07-19 09:34:06,759 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-07-19 09:34:06,773 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-07-19 09:34:09,237 INFO [watch_NEMO] running in process 5567 2024-07-19 09:34:09,239 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:34:09,240 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-07-19 09:34:09,241 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:34:09,501 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-07-19 09:39:09,838 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17796709 = 2024-07-20 05:12:00 UTC, 14.4% complete 2024-07-19 09:44:10,090 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17797220 = 2024-07-20 10:52:40 UTC, 30.2% complete 2024-07-19 09:49:09,784 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17797696 = 2024-07-20 16:10:00 UTC, 44.9% complete 2024-07-19 09:54:10,097 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17798197 = 2024-07-20 21:44:00 UTC, 60.4% complete 2024-07-19 09:54:24,537 INFO [download_results] nowcast 2024-07-19 results files from arbutus.cloud-nowcast downloaded 2024-07-19 09:54:24,542 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 09:54:24,554 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 09:54:24,559 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-07-18'], host='localhost') 2024-07-19 09:54:24,564 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 09:54:26,602 INFO [make_CHS_currents_file] running in process 687762 2024-07-19 09:54:26,603 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:54:26,604 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2024-07-19 09:54:26,605 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:54:28,044 INFO [make_plots] running in process 687759 2024-07-19 09:54:28,046 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:54:28,047 INFO [make_plots] publishing log messages to tcp://*:5566 2024-07-19 09:54:28,048 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:54:28,055 INFO [make_plots] running in process 687761 2024-07-19 09:54:28,072 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 09:54:28,089 INFO [make_plots] publishing log messages to tcp://*:5567 2024-07-19 09:54:28,096 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-07-19 09:54:28,691 INFO [make_plots] preparing render list for 2024-07-18 NEMO nowcast-blue comparison figures 2024-07-19 09:54:28,694 INFO [make_plots] preparing render list for 2024-07-19 NEMO nowcast-blue research figures 2024-07-19 09:54:29,771 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-blue research render list 2024-07-19 09:54:30,467 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-blue research render list 2024-07-19 09:54:30,468 INFO [make_plots] added Currents_sections_and_surface figure to 2024-07-19 NEMO nowcast-blue research render list 2024-07-19 09:54:30,469 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-07-19 NEMO nowcast-blue research render list 2024-07-19 09:54:30,470 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-07-19 NEMO nowcast-blue research render list 2024-07-19 09:54:30,471 INFO [make_plots] starting to render nemo nowcast research 19jul24 figures 2024-07-19 09:54:47,397 INFO [make_plots] added SH_wind figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,398 INFO [make_plots] added Compare_VENUS_East figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,398 INFO [make_plots] added Compare_VENUS_Central figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,399 INFO [make_plots] added Central_ADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,400 INFO [make_plots] added Central_depavADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,400 INFO [make_plots] added Central_timeavADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,401 INFO [make_plots] added East_ADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,402 INFO [make_plots] added East_depavADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,403 INFO [make_plots] added East_timeavADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,403 INFO [make_plots] added ddl_ADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,404 INFO [make_plots] added ddl_depavADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,405 INFO [make_plots] added ddl_timeavADCP figure to 2024-07-18 NEMO nowcast-blue comparison render list 2024-07-19 09:54:47,406 INFO [make_plots] starting to render nemo nowcast comparison 18jul24 figures 2024-07-19 09:54:56,699 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-19 for nowcast 2024-07-19 09:54:56,703 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-07-19 09:55:40,285 INFO [make_plots] finished rendering nemo nowcast research 19jul24 figures 2024-07-19 09:55:40,288 INFO [make_plots] nemo research plots for 2024-07-19 nowcast completed 2024-07-19 09:55:40,302 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-07-19 09:56:09,897 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available 2024-07-19 09:56:10,470 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-07-19 09:56:10,975 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-07-19 09:56:11,476 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available 2024-07-19 09:56:11,974 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-07-19 09:56:12,476 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-07-19 09:56:12,979 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available 2024-07-19 09:56:13,486 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-07-19 09:56:13,989 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-07-19 09:56:14,494 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available 2024-07-19 09:56:14,495 INFO [make_plots] finished rendering nemo nowcast comparison 18jul24 figures 2024-07-19 09:56:14,603 INFO [make_plots] nemo comparison plots for 2024-07-18 nowcast completed 2024-07-19 09:56:14,605 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-07-19 09:59:09,459 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17798667 = 2024-07-21 02:57:20 UTC, 74.9% complete 2024-07-19 10:04:09,459 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17799155 = 2024-07-21 08:22:40 UTC, 90.0% complete 2024-07-19 10:09:09,462 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed 2024-07-19 10:09:09,466 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-07-19 10:09:09,656 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast') 2024-07-19 10:09:09,664 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast') 2024-07-19 10:09:09,669 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:09:09,675 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:09:09,679 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:09:09,682 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:09:09,687 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:09:10,918 INFO [upload_forcing] running in process 687908 2024-07-19 10:09:10,920 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:10,921 INFO [upload_forcing] publishing log messages to tcp://*:5562 2024-07-19 10:09:10,922 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:10,923 INFO [upload_forcing] running in process 687911 2024-07-19 10:09:10,925 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:10,926 INFO [upload_forcing] publishing log messages to tcp://*:5566 2024-07-19 10:09:10,927 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:10,929 INFO [upload_forcing] running in process 687912 2024-07-19 10:09:10,931 INFO [upload_forcing] running in process 687910 2024-07-19 10:09:10,932 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:10,933 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-07-19 10:09:10,933 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:10,934 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:10,935 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-07-19 10:09:10,936 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:10,947 INFO [download_results] running in process 687913 2024-07-19 10:09:10,947 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:10,948 INFO [download_results] publishing log messages to tcp://*:5569 2024-07-19 10:09:10,949 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:10,950 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/19jul24 to /results/SalishSea/forecast.202111 2024-07-19 10:09:11,621 INFO [upload_forcing] turbidity 2024-07-19 forcing files upload to optimum-hindcast completed 2024-07-19 10:09:11,625 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-19 10:09:11,841 INFO [upload_forcing] turbidity 2024-07-19 forcing files upload to orcinus-nowcast-agrif completed 2024-07-19 10:09:11,845 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-19 10:09:11,863 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:09:11,898 INFO [upload_forcing] turbidity 2024-07-19 forcing files upload to arbutus.cloud-nowcast completed 2024-07-19 10:09:11,902 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-19 10:09:11,922 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:09:12,682 INFO [make_ww3_current_file] running in process 6210 2024-07-19 10:09:12,683 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:12,684 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2024-07-19 10:09:12,752 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:12,753 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-19 forecast run 2024-07-19 10:09:13,011 INFO [make_ww3_wind_file] running in process 6276 2024-07-19 10:09:13,012 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:13,013 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2024-07-19 10:09:13,014 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:13,015 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-19 forecast run 2024-07-19 10:09:13,030 INFO [make_forcing_links] running in process 687944 2024-07-19 10:09:13,031 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:13,032 INFO [make_forcing_links] publishing log messages to tcp://*:5562 2024-07-19 10:09:13,033 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:13,064 INFO [make_forcing_links] running in process 687946 2024-07-19 10:09:13,065 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:13,066 INFO [make_forcing_links] publishing log messages to tcp://*:5567 2024-07-19 10:09:13,067 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:13,851 INFO [make_forcing_links] nowcast-agrif 2024-07-19 forcing file links on orcinus-nowcast-agrif created 2024-07-19 10:09:13,855 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-07-19 10:09:13,875 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:09:14,181 INFO [make_forcing_links] nowcast-green 2024-07-19 forcing file links on arbutus.cloud-nowcast created 2024-07-19 10:09:14,185 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-07-19 10:09:14,204 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast') 2024-07-19 10:09:15,026 INFO [upload_forcing] turbidity 2024-07-19 forcing files upload to robot.graham completed 2024-07-19 10:09:15,029 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-19 10:09:15,281 INFO [run_NEMO_agrif] running in process 687960 2024-07-19 10:09:15,282 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:15,283 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5562 2024-07-19 10:09:15,284 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:16,353 INFO [run_NEMO] running in process 6593 2024-07-19 10:09:16,355 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:16,355 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-07-19 10:09:16,356 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:18,056 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/19jul24nowcast-green_2024-07-19T100916.384872-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-07-19 10:09:18,151 INFO [run_NEMO] nowcast-green NEMO run for 2024-07-19 on arbutus.cloud-nowcast started 2024-07-19 10:09:18,154 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-07-19 10:09:18,173 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast') 2024-07-19 10:09:21,002 INFO [watch_NEMO] running in process 7285 2024-07-19 10:09:21,003 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:21,004 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-07-19 10:09:21,005 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:21,280 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-07-19 10:09:24,990 INFO [run_NEMO_agrif] job id for 19jul24nowcast-agrif: 11101961.orca2.ibb 2024-07-19 10:09:24,991 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-07-19 queued on orcinus-nowcast-agrif 2024-07-19 10:09:24,996 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker 2024-07-19 10:09:25,014 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11101961.orca2.ibb'], host='localhost') 2024-07-19 10:09:25,175 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-19 forecast run 2024-07-19 10:09:25,203 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-07-19 10:09:26,159 INFO [watch_NEMO_agrif] running in process 687979 2024-07-19 10:09:26,162 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:09:26,163 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5562 2024-07-19 10:09:26,164 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:09:26,953 INFO [watch_NEMO_agrif] watching 19jul24nowcast-agrif job 11101961 on orcinus-nowcast-agrif 2024-07-19 10:09:27,186 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif job 11101961 is queued on orcinus-nowcast-agrif 2024-07-19 10:10:12,558 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-07-19 10:10:12,559 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-19 forecast run 2024-07-19 10:10:12,580 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast') 2024-07-19 10:10:15,572 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-07-19 10:10:15,593 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast') 2024-07-19 10:10:28,096 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch... 2024-07-19 10:14:21,365 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17794235 = 2024-07-19 01:42:40 UTC, 7.1% complete 2024-07-19 10:15:18,778 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete 2024-07-19 10:15:28,576 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7815059 = 2024-07-19 01:58:40 UTC, 8.2% complete 2024-07-19 10:19:21,268 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17794408 = 2024-07-19 03:38:00 UTC, 15.1% complete 2024-07-19 10:20:18,793 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete 2024-07-19 10:20:29,109 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7815264 = 2024-07-19 04:15:20 UTC, 17.7% complete 2024-07-19 10:24:21,414 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17794580 = 2024-07-19 05:32:40 UTC, 23.1% complete 2024-07-19 10:25:18,807 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 291, 67.4% complete 2024-07-19 10:25:29,571 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7815472 = 2024-07-19 06:34:00 UTC, 27.4% complete 2024-07-19 10:29:21,326 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17794753 = 2024-07-19 07:28:00 UTC, 31.1% complete 2024-07-19 10:30:18,794 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 387, 89.6% complete 2024-07-19 10:30:30,043 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7815681 = 2024-07-19 08:53:20 UTC, 37.1% complete 2024-07-19 10:34:21,798 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17794926 = 2024-07-19 09:23:20 UTC, 39.1% complete 2024-07-19 10:35:19,216 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed 2024-07-19 10:35:19,218 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-07-19 10:35:19,394 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:35:19,401 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast') 2024-07-19 10:35:20,717 INFO [download_wwatch3_results] running in process 688280 2024-07-19 10:35:20,718 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:35:20,720 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566 2024-07-19 10:35:20,721 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:35:26,488 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-07-19 10:35:26,507 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast') 2024-07-19 10:35:30,479 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7815887 = 2024-07-19 11:10:40 UTC, 46.6% complete 2024-07-19 10:35:33,416 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-07-19 10:39:22,324 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17795100 = 2024-07-19 11:19:20 UTC, 47.2% complete 2024-07-19 10:39:30,986 INFO [download_wwatch3_results] nowcast 2024-07-19 results files from arbutus.cloud-nowcast downloaded 2024-07-19 10:39:30,990 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-07-19 10:40:30,949 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7816095 = 2024-07-19 13:29:20 UTC, 56.2% complete 2024-07-19 10:40:33,222 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 84, 13.0% complete 2024-07-19 10:41:08,093 INFO [download_results] forecast 2024-07-19 results files from arbutus.cloud-nowcast downloaded 2024-07-19 10:41:08,097 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 10:41:08,117 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:41:10,100 INFO [make_CHS_currents_file] running in process 688306 2024-07-19 10:41:10,101 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:41:10,102 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5566 2024-07-19 10:41:10,104 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:41:50,733 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-19 for forecast 2024-07-19 10:41:50,736 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-07-19 10:41:50,758 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:41:51,850 INFO [update_forecast_datasets] running in process 688366 2024-07-19 10:41:51,851 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:41:51,852 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566 2024-07-19 10:41:51,853 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:41:51,854 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/19jul24 run 2024-07-19 10:41:52,293 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/19jul24 run: /results/SalishSea/rolling-forecasts/nemo 2024-07-19 10:41:52,294 INFO [update_forecast_datasets] nemo 2024-07-19 forecast rolling forecast datasets updated 2024-07-19 10:41:52,296 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-07-19 10:41:52,316 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-07-19 10:41:52,320 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:41:52,325 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 10:41:53,478 INFO [ping_erddap] running in process 688371 2024-07-19 10:41:53,479 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:41:53,480 INFO [ping_erddap] publishing log messages to tcp://*:5566 2024-07-19 10:41:53,481 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:41:53,490 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-07-19 10:41:53,493 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-19 10:41:54,590 INFO [make_surface_current_tiles] running in process 688374 2024-07-19 10:41:54,592 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:41:54,592 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5566 2024-07-19 10:41:54,593 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:41:54,620 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-19 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/19jul24 2024-07-19 10:41:56,458 INFO [make_plots] running in process 688373 2024-07-19 10:41:56,459 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:41:56,460 INFO [make_plots] publishing log messages to tcp://*:5567 2024-07-19 10:41:56,461 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:41:57,065 INFO [make_plots] preparing render list for 2024-07-19 NEMO forecast publish figures 2024-07-19 10:41:57,185 INFO [make_plots] added Website_thumbnail figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,186 INFO [make_plots] added Threshold_website figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,187 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,188 INFO [make_plots] added Boundary Bay figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,189 INFO [make_plots] added Campbell River figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,190 INFO [make_plots] added Cherry Point figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,190 INFO [make_plots] added Friday Harbor figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,191 INFO [make_plots] added Halfmoon Bay figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,192 INFO [make_plots] added Nanaimo figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,192 INFO [make_plots] added Neah Bay figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,193 INFO [make_plots] added New Westminster figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,194 INFO [make_plots] added Patricia Bay figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,195 INFO [make_plots] added Point Atkinson figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,196 INFO [make_plots] added Port Renfrew figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,197 INFO [make_plots] added Sand Heads figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,198 INFO [make_plots] added Sandy Cove figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,199 INFO [make_plots] added Squamish figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,199 INFO [make_plots] added Victoria figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,200 INFO [make_plots] added Woodwards Landing figure to 2024-07-19 NEMO forecast publish render list 2024-07-19 10:41:57,201 INFO [make_plots] starting to render nemo forecast publish 19jul24 figures 2024-07-19 10:44:21,246 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17795272 = 2024-07-19 13:14:00 UTC, 55.2% complete 2024-07-19 10:45:31,419 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7816304 = 2024-07-19 15:48:40 UTC, 65.9% complete 2024-07-19 10:45:33,072 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 180, 27.8% complete 2024-07-19 10:49:21,247 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17795443 = 2024-07-19 15:08:00 UTC, 63.1% complete 2024-07-19 10:50:31,854 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7816510 = 2024-07-19 18:06:00 UTC, 75.5% complete 2024-07-19 10:50:33,073 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 273, 42.1% complete 2024-07-19 10:52:27,044 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 10:52:27,214 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-01-06'], host='localhost') 2024-07-19 10:52:27,220 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 10:52:27,224 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 10:52:28,447 INFO [download_results] running in process 690215 2024-07-19 10:52:28,448 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 10:52:28,449 INFO [download_results] publishing log messages to tcp://*:5564 2024-07-19 10:52:28,450 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 10:52:28,451 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06jan18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 10:52:32,714 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 10:52:32,801 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/19jul24 2024-07-19 10:53:08,179 INFO [download_results] hindcast 2018-01-06 results files from optimum-hindcast downloaded 2024-07-19 10:53:08,187 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 10:53:08,206 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-01-06'], host='localhost') 2024-07-19 10:53:09,642 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 10:54:21,250 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17795615 = 2024-07-19 17:02:40 UTC, 71.1% complete 2024-07-19 10:54:49,282 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-07-19 10:54:49,283 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-07-19 10:55:24,598 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 10:55:32,330 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7816718 = 2024-07-19 20:24:40 UTC, 85.1% complete 2024-07-19 10:55:33,076 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 363, 56.0% complete 2024-07-19 10:57:07,845 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/19jul24 2024-07-19 10:57:07,857 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-19 forecast into /results/nowcast-sys/figures/surface_currents/forecast/19jul24 2024-07-19 10:57:07,858 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-19 forecast completed 2024-07-19 10:57:08,174 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-07-19 10:57:11,371 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 10:57:11,740 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 10:57:46,609 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 10:58:56,295 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 10:59:21,254 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17795788 = 2024-07-19 18:58:00 UTC, 79.1% complete 2024-07-19 10:59:31,881 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:00:07,766 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:00:32,800 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7816926 = 2024-07-19 22:43:20 UTC, 94.7% complete 2024-07-19 11:00:33,079 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 453, 69.9% complete 2024-07-19 11:00:43,223 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:01:18,139 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:01:54,094 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:02:29,634 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:02:29,987 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:03:04,751 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:03:39,483 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-14 00:00:00Z to 2024-07-21 11:55:00Z 2024-07-19 11:04:14,982 INFO [make_plots] finished rendering nemo forecast publish 19jul24 figures 2024-07-19 11:04:14,984 INFO [make_plots] nemo publish plots for 2024-07-19 forecast completed 2024-07-19 11:04:14,991 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-07-19 11:04:15,159 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:04:17,552 INFO [make_feeds] running in process 690359 2024-07-19 11:04:17,555 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:04:17,556 INFO [make_feeds] publishing log messages to tcp://*:5564 2024-07-19 11:04:17,557 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:04:21,259 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17795961 = 2024-07-19 20:53:20 UTC, 87.1% complete 2024-07-19 11:04:45,586 INFO [make_feeds] ATOM feeds for 2024-07-19 forecast run completed 2024-07-19 11:04:45,589 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-07-19 11:05:33,082 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 549, 84.7% complete 2024-07-19 11:05:33,226 INFO [watch_NEMO_agrif] 19jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7817040 = 2024-07-19 23:59:20 UTC, 100.0% complete 2024-07-19 11:09:21,259 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17796132 = 2024-07-19 22:47:20 UTC, 95.0% complete 2024-07-19 11:10:33,084 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 642, 99.1% complete 2024-07-19 11:10:33,440 INFO [watch_NEMO_agrif] NEMO AGRIF run 11101961.orca2.ibb on orcinus-nowcast-agrif completed 2024-07-19 11:10:33,445 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker 2024-07-19 11:10:33,769 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:10:34,936 INFO [download_results] running in process 690397 2024-07-19 11:10:34,937 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:10:34,938 INFO [download_results] publishing log messages to tcp://*:5562 2024-07-19 11:10:34,939 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:10:34,940 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/19jul24 to /results/SalishSea/nowcast-agrif.201702 2024-07-19 11:10:58,044 INFO [download_results] nowcast-agrif 2024-07-19 results files from orcinus-nowcast-agrif downloaded 2024-07-19 11:10:58,052 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 11:10:58,223 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:11:01,143 INFO [make_plots] running in process 690406 2024-07-19 11:11:01,146 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:11:01,147 INFO [make_plots] publishing log messages to tcp://*:5562 2024-07-19 11:11:01,148 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:11:01,661 INFO [make_plots] preparing render list for 2024-07-19 NEMO nowcast-agrif research figures 2024-07-19 11:11:01,662 INFO [make_plots] added baynes_sound_surface figure to 2024-07-19 NEMO nowcast-agrif research render list 2024-07-19 11:11:01,663 INFO [make_plots] starting to render nemo nowcast-agrif research 19jul24 figures 2024-07-19 11:11:10,778 INFO [make_plots] finished rendering nemo nowcast-agrif research 19jul24 figures 2024-07-19 11:11:10,781 INFO [make_plots] nemo research plots for 2024-07-19 nowcast-agrif completed 2024-07-19 11:11:10,785 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-07-19 11:14:21,261 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17796240 = 2024-07-19 23:59:20 UTC, 100.0% complete 2024-07-19 11:15:33,087 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed 2024-07-19 11:15:33,089 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-07-19 11:15:33,402 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:15:34,479 INFO [download_wwatch3_results] running in process 690461 2024-07-19 11:15:34,480 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:15:34,481 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562 2024-07-19 11:15:34,482 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:18:04,597 INFO [download_wwatch3_results] forecast 2024-07-19 results files from arbutus.cloud-nowcast downloaded 2024-07-19 11:18:04,601 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-07-19 11:18:04,770 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:18:05,852 INFO [update_forecast_datasets] running in process 690519 2024-07-19 11:18:05,854 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:18:05,854 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2024-07-19 11:18:05,855 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:18:05,856 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/19jul24 run 2024-07-19 11:18:05,858 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/19jul24 run 2024-07-19 11:18:05,899 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/19jul24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-07-19 11:18:05,900 INFO [update_forecast_datasets] wwatch3 2024-07-19 forecast rolling forecast datasets updated 2024-07-19 11:18:05,903 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-07-19 11:18:06,077 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-07-19 11:18:07,049 INFO [ping_erddap] running in process 690524 2024-07-19 11:18:07,050 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:18:07,050 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-07-19 11:18:07,051 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:18:07,053 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-07-19 11:18:07,054 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-19 11:18:07,196 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:18:10,352 INFO [make_plots] running in process 690529 2024-07-19 11:18:10,354 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:18:10,354 INFO [make_plots] publishing log messages to tcp://*:5562 2024-07-19 11:18:10,355 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:18:10,356 INFO [make_plots] preparing render list for 2024-07-19 WaveWatch3 forecast publish figures 2024-07-19 11:18:10,357 INFO [make_plots] added Halibut Bank figure to 2024-07-19 WaveWatch3 forecast publish render list 2024-07-19 11:18:10,358 INFO [make_plots] added Sentry Shoal figure to 2024-07-19 WaveWatch3 forecast publish render list 2024-07-19 11:18:10,359 INFO [make_plots] starting to render wwatch3 forecast publish 19jul24 figures 2024-07-19 11:18:10,632 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-07-19 11:18:18,000 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-07-19 11:18:23,992 INFO [make_plots] finished rendering wwatch3 forecast publish 19jul24 figures 2024-07-19 11:18:23,992 INFO [make_plots] wwatch3 publish plots for 2024-07-19 forecast completed 2024-07-19 11:18:23,996 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-07-19 11:19:21,269 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed 2024-07-19 11:19:21,273 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-07-19 11:19:21,599 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:19:22,762 INFO [download_results] running in process 690565 2024-07-19 11:19:22,764 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:19:22,765 INFO [download_results] publishing log messages to tcp://*:5562 2024-07-19 11:19:22,766 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:19:22,767 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/19jul24 to /results2/SalishSea/nowcast-green.202111 2024-07-19 11:53:42,525 INFO [download_results] nowcast-green 2024-07-19 results files from arbutus.cloud-nowcast downloaded 2024-07-19 11:53:42,531 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 11:53:42,706 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:53:42,713 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost') 2024-07-19 11:53:42,717 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:53:42,722 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:53:42,727 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-07-19'], host='localhost') 2024-07-19 11:53:44,032 INFO [ping_erddap] running in process 690644 2024-07-19 11:53:44,034 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:53:44,035 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-07-19 11:53:44,036 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:53:44,041 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created 2024-07-19 11:53:44,043 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-19 11:53:45,129 INFO [make_averaged_dataset] running in process 690647 2024-07-19 11:53:45,131 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:53:45,132 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562 2024-07-19 11:53:45,133 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:53:45,134 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Jul-2024 physics 2024-07-19 11:53:45,135 INFO [make_averaged_dataset] running in process 690646 2024-07-19 11:53:45,136 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:53:45,137 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564 2024-07-19 11:53:45,138 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:53:45,140 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Jul-2024 chemistry 2024-07-19 11:53:45,141 INFO [make_averaged_dataset] running in process 690645 2024-07-19 11:53:45,142 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:53:45,143 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566 2024-07-19 11:53:45,144 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:53:45,145 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Jul-2024 biology 2024-07-19 11:53:46,265 INFO [make_plots] running in process 690642 2024-07-19 11:53:46,266 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 11:53:46,267 INFO [make_plots] publishing log messages to tcp://*:5567 2024-07-19 11:53:46,268 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-07-19 11:53:46,761 INFO [make_plots] preparing render list for 2024-07-19 NEMO nowcast-green research figures 2024-07-19 11:53:48,426 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:49,138 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:49,877 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:50,640 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:51,389 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:52,123 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:52,847 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:53,677 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:54,508 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:55,313 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:56,135 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:56,868 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:57,586 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:57,587 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:57,588 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:57,589 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-07-19 NEMO nowcast-green research render list 2024-07-19 11:53:57,589 INFO [make_plots] starting to render nemo nowcast-green research 19jul24 figures 2024-07-19 11:54:23,795 INFO [make_averaged_dataset] day-averaged dataset for 19-Jul-2024 physics created 2024-07-19 11:54:23,798 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-07-19 11:54:59,985 INFO [make_averaged_dataset] day-averaged dataset for 19-Jul-2024 chemistry created 2024-07-19 11:54:59,989 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-07-19 11:55:14,152 INFO [make_averaged_dataset] day-averaged dataset for 19-Jul-2024 biology created 2024-07-19 11:55:14,155 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker 2024-07-19 12:17:37,049 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 12:17:37,382 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-01-11'], host='localhost') 2024-07-19 12:17:37,388 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 12:17:37,392 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 12:17:38,996 INFO [download_results] running in process 690867 2024-07-19 12:17:38,998 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 12:17:38,999 INFO [download_results] publishing log messages to tcp://*:5566 2024-07-19 12:17:39,000 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 12:17:39,001 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11jan18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 12:17:43,073 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 12:18:26,737 INFO [download_results] hindcast 2018-01-11 results files from optimum-hindcast downloaded 2024-07-19 12:18:26,747 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 12:18:26,922 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-01-11'], host='localhost') 2024-07-19 12:18:28,470 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 13:04:47,265 INFO [make_plots] finished rendering nemo nowcast-green research 19jul24 figures 2024-07-19 13:04:47,433 INFO [make_plots] nemo research plots for 2024-07-19 nowcast-green completed 2024-07-19 13:04:47,502 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-07-19 13:47:47,858 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 13:47:48,263 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-01-16'], host='localhost') 2024-07-19 13:47:48,270 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 13:47:48,275 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 13:47:49,746 INFO [download_results] running in process 691581 2024-07-19 13:47:49,747 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 13:47:49,772 INFO [download_results] publishing log messages to tcp://*:5566 2024-07-19 13:47:49,774 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 13:47:49,775 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16jan18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 13:47:54,206 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 13:48:34,880 INFO [download_results] hindcast 2018-01-16 results files from optimum-hindcast downloaded 2024-07-19 13:48:34,888 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 13:48:35,105 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-01-16'], host='localhost') 2024-07-19 13:48:36,568 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 15:06:19,680 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240719/18/ 2024-07-19 15:06:19,681 INFO [collect_weather] 2024-07-19 2.5km weather forecast 18 collection complete 2024-07-19 15:06:19,686 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-07-19 15:06:20,029 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost') 2024-07-19 15:06:20,037 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost') 2024-07-19 15:06:20,041 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost') 2024-07-19 15:06:20,047 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-07-20'], host='localhost') 2024-07-19 15:06:20,716 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240719/18/ 2024-07-19 15:06:20,717 INFO [crop_gribs] 2024-07-19 18 GRIBs cropping complete 2024-07-19 15:06:20,719 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-07-19 15:06:22,068 INFO [download_weather] running in process 692335 2024-07-19 15:06:22,069 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 15:06:22,070 INFO [download_weather] publishing log messages to tcp://*:5564 2024-07-19 15:06:22,072 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-07-19 15:06:22,073 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240719 2024-07-19 15:06:22,074 INFO [collect_weather] running in process 692338 2024-07-19 15:06:22,075 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 15:06:22,076 INFO [collect_weather] publishing log messages to tcp://*:5565 2024-07-19 15:06:22,077 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-07-19 15:06:22,079 INFO [download_weather] running in process 692337 2024-07-19 15:06:22,079 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 15:06:22,081 INFO [download_weather] publishing log messages to tcp://*:5566 2024-07-19 15:06:22,081 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555 2024-07-19 15:06:22,082 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240719 2024-07-19 15:06:22,180 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/ 2024-07-19 15:06:23,405 INFO [crop_gribs] running in process 692340 2024-07-19 15:06:23,406 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 15:06:23,407 INFO [crop_gribs] publishing log messages to tcp://*:5567 2024-07-19 15:06:23,408 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-07-19 15:06:23,409 INFO [crop_gribs] cropping 2024-07-20 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain 2024-07-19 15:06:23,424 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240720/00/ 2024-07-19 15:10:58,689 INFO [download_weather] 2024-07-19 1km weather forecast 00 downloads complete 2024-07-19 15:10:58,693 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-07-19 15:11:19,653 INFO [download_weather] 2024-07-19 1km weather forecast 12 downloads complete 2024-07-19 15:11:19,656 INFO [manager] checklist updated with [weather forecast] items from download_weather worker 2024-07-19 15:12:58,467 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 15:12:58,819 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-01-21'], host='localhost') 2024-07-19 15:12:58,827 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 15:12:58,832 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 15:13:00,177 INFO [download_results] running in process 692459 2024-07-19 15:13:00,178 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 15:13:00,179 INFO [download_results] publishing log messages to tcp://*:5566 2024-07-19 15:13:00,180 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 15:13:00,181 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21jan18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 15:13:04,396 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 15:13:50,391 INFO [download_results] hindcast 2018-01-21 results files from optimum-hindcast downloaded 2024-07-19 15:13:50,402 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 15:13:50,606 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-01-21'], host='localhost') 2024-07-19 15:13:51,689 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 16:53:09,943 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 16:53:10,295 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-01-26'], host='localhost') 2024-07-19 16:53:10,303 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 16:53:10,308 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 16:53:11,675 INFO [download_results] running in process 692975 2024-07-19 16:53:11,676 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 16:53:11,677 INFO [download_results] publishing log messages to tcp://*:5562 2024-07-19 16:53:11,678 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 16:53:11,679 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jan18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 16:53:15,757 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 16:53:56,578 INFO [download_results] hindcast 2018-01-26 results files from optimum-hindcast downloaded 2024-07-19 16:53:56,587 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 16:53:56,791 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-01-26'], host='localhost') 2024-07-19 16:53:57,909 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 16:53:58,115 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-jan', 'robot.graham'], host='localhost') 2024-07-19 16:53:59,132 INFO [archive_tarball] running in process 693019 2024-07-19 16:53:59,133 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 16:53:59,134 INFO [archive_tarball] publishing log messages to tcp://*:5562 2024-07-19 16:53:59,135 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-07-19 16:53:59,135 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jan18.tar from /ocean/sallen/timbucktwo/oxygen/*jan18/ 2024-07-19 16:59:42,034 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jan18.index from /ocean/dlatorne/oxygen-jan18.tar 2024-07-19 16:59:42,110 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jan18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-07-19 17:06:48,727 INFO [archive_tarball] hindcast *jan18 results files archived to robot.graham 2024-07-19 17:06:48,730 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-07-19 18:23:20,632 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 18:23:20,990 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-02-01'], host='localhost') 2024-07-19 18:23:20,998 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 18:23:21,004 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 18:23:22,342 INFO [download_results] running in process 693396 2024-07-19 18:23:22,343 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 18:23:22,345 INFO [download_results] publishing log messages to tcp://*:5562 2024-07-19 18:23:22,346 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 18:23:22,347 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01feb18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 18:23:26,474 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 18:24:02,776 INFO [download_results] hindcast 2018-02-01 results files from optimum-hindcast downloaded 2024-07-19 18:24:02,786 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 18:24:02,983 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-02-01'], host='localhost') 2024-07-19 18:24:04,096 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 19:48:31,112 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 19:48:31,471 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-02-06'], host='localhost') 2024-07-19 19:48:31,478 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 19:48:31,482 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 19:48:32,656 INFO [download_results] running in process 693717 2024-07-19 19:48:32,657 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 19:48:32,658 INFO [download_results] publishing log messages to tcp://*:5562 2024-07-19 19:48:32,659 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 19:48:32,660 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06feb18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 19:48:39,548 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 19:49:12,451 INFO [download_results] hindcast 2018-02-06 results files from optimum-hindcast downloaded 2024-07-19 19:49:12,461 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 19:49:12,667 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-02-06'], host='localhost') 2024-07-19 19:49:13,730 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 21:03:34,904 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240720/00/ 2024-07-19 21:03:34,905 INFO [collect_weather] 2024-07-20 2.5km weather forecast 00 collection complete 2024-07-19 21:03:34,909 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-07-19 21:03:35,088 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240720/00/ 2024-07-19 21:03:35,089 INFO [crop_gribs] 2024-07-20 00 GRIBs cropping complete 2024-07-19 21:03:35,264 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost') 2024-07-19 21:03:35,270 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-07-20'], host='localhost') 2024-07-19 21:03:35,276 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-07-19 21:03:36,548 INFO [collect_weather] running in process 694229 2024-07-19 21:03:36,549 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 21:03:36,550 INFO [collect_weather] publishing log messages to tcp://*:5562 2024-07-19 21:03:36,551 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-07-19 21:03:36,570 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/ 2024-07-19 21:03:37,924 INFO [crop_gribs] running in process 694231 2024-07-19 21:03:37,925 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 21:03:37,926 INFO [crop_gribs] publishing log messages to tcp://*:5564 2024-07-19 21:03:37,926 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-07-19 21:03:37,927 INFO [crop_gribs] cropping 2024-07-20 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain 2024-07-19 21:03:37,943 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240720/06/ 2024-07-19 21:13:41,232 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 21:13:41,577 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-02-11'], host='localhost') 2024-07-19 21:13:41,584 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 21:13:41,590 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 21:13:42,764 INFO [download_results] running in process 694288 2024-07-19 21:13:42,765 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 21:13:42,766 INFO [download_results] publishing log messages to tcp://*:5567 2024-07-19 21:13:42,767 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 21:13:42,768 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11feb18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 21:13:47,209 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 21:14:22,448 INFO [download_results] hindcast 2018-02-11 results files from optimum-hindcast downloaded 2024-07-19 21:14:22,457 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 21:14:22,659 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-02-11'], host='localhost') 2024-07-19 21:14:23,698 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-19 22:43:52,252 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-19 22:43:52,600 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-02-16'], host='localhost') 2024-07-19 22:43:52,607 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 22:43:52,612 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-19 22:43:53,754 INFO [download_results] running in process 694746 2024-07-19 22:43:53,755 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-19 22:43:53,756 INFO [download_results] publishing log messages to tcp://*:5565 2024-07-19 22:43:53,757 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-19 22:43:53,758 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16feb18 to /ocean/sallen/timbucktwo/oxygen 2024-07-19 22:43:58,228 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-19 22:44:36,514 INFO [download_results] hindcast 2018-02-16 results files from optimum-hindcast downloaded 2024-07-19 22:44:36,525 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-19 22:44:36,729 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-02-16'], host='localhost') 2024-07-19 22:44:37,881 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-20 00:09:02,338 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-20 00:09:02,678 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-02-21'], host='localhost') 2024-07-20 00:09:02,684 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-20 00:09:02,689 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-20 00:09:03,836 INFO [download_results] running in process 695215 2024-07-20 00:09:03,837 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 00:09:03,838 INFO [download_results] publishing log messages to tcp://*:5566 2024-07-20 00:09:03,839 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-20 00:09:03,840 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21feb18 to /ocean/sallen/timbucktwo/oxygen 2024-07-20 00:09:08,153 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-20 00:09:44,671 INFO [download_results] hindcast 2018-02-21 results files from optimum-hindcast downloaded 2024-07-20 00:09:44,682 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-20 00:09:44,887 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-02-21'], host='localhost') 2024-07-20 00:09:46,029 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-20 01:04:09,518 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-20 01:04:09,882 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-02-26'], host='localhost') 2024-07-20 01:04:09,888 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-20 01:04:09,893 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-20 01:04:11,071 INFO [download_results] running in process 695546 2024-07-20 01:04:11,072 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 01:04:11,073 INFO [download_results] publishing log messages to tcp://*:5567 2024-07-20 01:04:11,075 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-20 01:04:11,076 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26feb18 to /ocean/sallen/timbucktwo/oxygen 2024-07-20 01:04:15,584 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-20 01:04:40,962 INFO [download_results] hindcast 2018-02-26 results files from optimum-hindcast downloaded 2024-07-20 01:04:40,969 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-20 01:04:41,169 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-02-26'], host='localhost') 2024-07-20 01:04:42,195 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-20 01:04:42,401 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-feb', 'robot.graham'], host='localhost') 2024-07-20 01:04:43,410 INFO [archive_tarball] running in process 695593 2024-07-20 01:04:43,411 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 01:04:43,412 INFO [archive_tarball] publishing log messages to tcp://*:5565 2024-07-20 01:04:43,413 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555 2024-07-20 01:04:43,413 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb18.tar from /ocean/sallen/timbucktwo/oxygen/*feb18/ 2024-07-20 01:10:06,110 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb18.index from /ocean/dlatorne/oxygen-feb18.tar 2024-07-20 01:10:06,193 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-feb18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/ 2024-07-20 01:16:42,515 INFO [archive_tarball] hindcast *feb18 results files archived to robot.graham 2024-07-20 01:16:42,518 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker 2024-07-20 02:29:19,650 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-20 02:29:19,993 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-03-01'], host='localhost') 2024-07-20 02:29:20,000 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-20 02:29:20,004 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-20 02:29:21,168 INFO [download_results] running in process 696026 2024-07-20 02:29:21,169 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 02:29:21,170 INFO [download_results] publishing log messages to tcp://*:5567 2024-07-20 02:29:21,172 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-20 02:29:21,172 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01mar18 to /ocean/sallen/timbucktwo/oxygen 2024-07-20 02:29:25,625 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-20 02:30:01,009 INFO [download_results] hindcast 2018-03-01 results files from optimum-hindcast downloaded 2024-07-20 02:30:01,019 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-20 02:30:01,222 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-03-01'], host='localhost') 2024-07-20 02:30:02,276 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-20 03:03:27,312 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240720/06/ 2024-07-20 03:03:27,313 INFO [collect_weather] 2024-07-20 2.5km weather forecast 06 collection complete 2024-07-20 03:03:27,316 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker 2024-07-20 03:03:27,663 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,669 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,675 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,679 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,682 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,687 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,691 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,695 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,705 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,709 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,715 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,728 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,739 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,745 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,760 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost') 2024-07-20 03:03:27,770 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost') 2024-07-20 03:03:27,783 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost') 2024-07-20 03:03:27,796 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-07-19'], host='localhost') 2024-07-20 03:03:27,810 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost') 2024-07-20 03:03:27,836 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost') 2024-07-20 03:03:27,848 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost') 2024-07-20 03:03:29,323 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240720/06/ 2024-07-20 03:03:29,324 INFO [crop_gribs] 2024-07-20 06 GRIBs cropping complete 2024-07-20 03:03:29,326 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker 2024-07-20 03:03:29,958 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost') 2024-07-20 03:03:30,928 INFO [collect_NeahBay_ssh] running in process 696169 2024-07-20 03:03:30,929 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:30,930 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562 2024-07-20 03:03:30,930 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:30,931 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-07-20 2024-07-20 03:03:30,951 INFO [collect_weather] running in process 696170 2024-07-20 03:03:30,952 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:30,953 INFO [collect_weather] publishing log messages to tcp://*:5564 2024-07-20 03:03:30,954 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:30,976 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/ 2024-07-20 03:03:31,946 INFO [collect_river_data] running in process 696157 2024-07-20 03:03:31,949 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:31,950 INFO [collect_river_data] publishing log messages to tcp://*:5566 2024-07-20 03:03:31,951 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:31,953 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-07-19 2024-07-20 03:03:32,011 INFO [collect_river_data] running in process 696151 2024-07-20 03:03:32,015 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,015 INFO [collect_river_data] publishing log messages to tcp://*:5567 2024-07-20 03:03:32,017 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,018 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-07-19 2024-07-20 03:03:32,023 INFO [collect_river_data] running in process 696162 2024-07-20 03:03:32,026 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,027 INFO [collect_river_data] publishing log messages to tcp://*:5568 2024-07-20 03:03:32,029 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,029 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-07-19 2024-07-20 03:03:32,030 INFO [collect_river_data] running in process 696153 2024-07-20 03:03:32,031 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,032 INFO [collect_river_data] publishing log messages to tcp://*:5569 2024-07-20 03:03:32,033 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,034 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-07-19 2024-07-20 03:03:32,035 INFO [collect_river_data] running in process 696155 2024-07-20 03:03:32,036 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,037 INFO [collect_river_data] publishing log messages to tcp://*:5570 2024-07-20 03:03:32,037 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,038 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-07-19 2024-07-20 03:03:32,039 INFO [collect_river_data] running in process 696156 2024-07-20 03:03:32,040 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,041 INFO [collect_river_data] publishing log messages to tcp://*:5571 2024-07-20 03:03:32,042 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,042 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-07-19 2024-07-20 03:03:32,047 INFO [collect_river_data] running in process 696152 2024-07-20 03:03:32,053 INFO [collect_river_data] running in process 696149 2024-07-20 03:03:32,054 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,055 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,056 INFO [collect_river_data] publishing log messages to tcp://*:5572 2024-07-20 03:03:32,058 INFO [collect_river_data] publishing log messages to tcp://*:5573 2024-07-20 03:03:32,059 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,060 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,061 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-07-19 2024-07-20 03:03:32,061 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-07-19 2024-07-20 03:03:32,107 INFO [collect_river_data] running in process 696160 2024-07-20 03:03:32,108 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,109 INFO [collect_river_data] publishing log messages to tcp://*:5574 2024-07-20 03:03:32,109 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,110 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-07-19 2024-07-20 03:03:32,151 INFO [collect_river_data] running in process 696161 2024-07-20 03:03:32,152 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,153 INFO [collect_river_data] publishing log messages to tcp://*:5575 2024-07-20 03:03:32,154 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,155 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-07-19 2024-07-20 03:03:32,181 INFO [collect_river_data] running in process 696154 2024-07-20 03:03:32,182 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,183 INFO [collect_river_data] publishing log messages to tcp://*:5576 2024-07-20 03:03:32,185 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,186 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-07-19 2024-07-20 03:03:32,190 INFO [collect_river_data] running in process 696164 2024-07-20 03:03:32,191 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,191 INFO [collect_river_data] publishing log messages to tcp://*:5577 2024-07-20 03:03:32,193 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,194 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-07-19 2024-07-20 03:03:32,202 INFO [collect_river_data] running in process 696159 2024-07-20 03:03:32,203 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,204 INFO [collect_river_data] publishing log messages to tcp://*:5578 2024-07-20 03:03:32,206 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,208 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-07-19 2024-07-20 03:03:32,210 INFO [collect_river_data] running in process 696163 2024-07-20 03:03:32,210 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,211 INFO [collect_river_data] publishing log messages to tcp://*:5579 2024-07-20 03:03:32,212 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,213 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-07-19 2024-07-20 03:03:32,544 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/RobertsCreek_flow 2024-07-20 03:03:32,545 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-07-19 completed 2024-07-20 03:03:32,547 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow 2024-07-20 03:03:32,548 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:32,551 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-07-19 completed 2024-07-20 03:03:32,689 INFO [get_vfpa_hadcp] running in process 696168 2024-07-20 03:03:32,690 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,691 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5590 2024-07-20 03:03:32,692 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,693 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-19 2024-07-20 03:03:32,698 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow 2024-07-20 03:03:32,700 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-07-19 completed 2024-07-20 03:03:32,704 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow 2024-07-20 03:03:32,706 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-07-19 completed 2024-07-20 03:03:32,740 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Homathko_Mouth_flow 2024-07-20 03:03:32,740 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-07-19 completed 2024-07-20 03:03:32,744 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-07-19 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow 2024-07-20 03:03:32,745 INFO [collect_river_data] ECCC Englishman river data collection for 2024-07-19 completed 2024-07-20 03:03:32,764 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow 2024-07-20 03:03:32,766 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-07-19 completed 2024-07-20 03:03:32,778 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-07-19 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow 2024-07-20 03:03:32,779 INFO [collect_river_data] ECCC Capilano river data collection for 2024-07-19 completed 2024-07-20 03:03:32,851 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow 2024-07-20 03:03:32,852 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-07-19 completed 2024-07-20 03:03:32,867 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow 2024-07-20 03:03:32,868 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-07-19 completed 2024-07-20 03:03:32,869 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-07-19 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow 2024-07-20 03:03:32,870 INFO [collect_river_data] ECCC Fraser river data collection for 2024-07-19 completed 2024-07-20 03:03:32,909 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow 2024-07-20 03:03:32,909 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-07-19 completed 2024-07-20 03:03:32,917 INFO [crop_gribs] running in process 696171 2024-07-20 03:03:32,918 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:32,918 INFO [crop_gribs] publishing log messages to tcp://*:5591 2024-07-20 03:03:32,919 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:32,920 INFO [crop_gribs] cropping 2024-07-20 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain 2024-07-20 03:03:32,937 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow 2024-07-20 03:03:32,938 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-07-19 completed 2024-07-20 03:03:32,939 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240720/12/ 2024-07-20 03:03:33,016 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-07-19 to: /results/forcing/rivers/observations/Salmon_Sayward_flow 2024-07-20 03:03:33,017 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-07-19 completed 2024-07-20 03:03:33,041 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:33,272 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:33,300 INFO [collect_NeahBay_ssh] 2024-07-20T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete 2024-07-20 03:03:33,465 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:33,491 INFO [get_onc_ferry] running in process 696167 2024-07-20 03:03:33,492 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:33,493 INFO [get_onc_ferry] publishing log messages to tcp://*:5566 2024-07-20 03:03:33,494 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:33,495 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-07-19 2024-07-20 03:03:33,524 INFO [get_onc_ctd] running in process 696165 2024-07-20 03:03:33,525 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:33,526 INFO [get_onc_ctd] publishing log messages to tcp://*:5571 2024-07-20 03:03:33,527 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:33,528 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-07-19 2024-07-20 03:03:33,544 INFO [get_onc_ctd] running in process 696166 2024-07-20 03:03:33,545 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:33,547 INFO [get_onc_ctd] publishing log messages to tcp://*:5592 2024-07-20 03:03:33,548 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:33,548 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-07-19 2024-07-20 03:03:33,653 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:33,756 INFO [grib_to_netcdf] running in process 696438 2024-07-20 03:03:33,757 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:33,758 INFO [grib_to_netcdf] publishing log messages to tcp://*:5574 2024-07-20 03:03:33,759 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:33,759 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-20 forecast2 run 2024-07-20 03:03:33,868 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:34,056 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:34,243 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:34,432 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:34,669 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:34,901 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:34,919 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-19 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202407.nc 2024-07-20 03:03:34,920 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-07 netcdf file 2024-07-20 03:03:35,079 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:35,255 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:35,430 INFO [manager] checklist updated with [river data] items from collect_river_data worker 2024-07-20 03:03:35,585 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-07-19 2024-07-20 03:03:35,605 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker 2024-07-20 03:03:35,788 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-07-20'], host='localhost') 2024-07-20 03:03:35,794 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker 2024-07-20 03:03:36,095 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost') 2024-07-20 03:03:36,156 CRITICAL [get_onc_ctd] 2024-07-19 ONC SCVIP CTD T&S file creation failed 2024-07-20 03:03:37,258 INFO [ping_erddap] running in process 696637 2024-07-20 03:03:37,259 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:37,260 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-07-20 03:03:37,260 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:37,262 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created 2024-07-20 03:03:37,264 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-20 03:03:38,696 INFO [make_ssh_files] running in process 696624 2024-07-20 03:03:38,698 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:38,698 INFO [make_ssh_files] publishing log messages to tcp://*:5562 2024-07-20 03:03:38,699 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:38,700 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-07-20 NOAA Neah Bay 00Z observation and forecast values 2024-07-20 03:03:42,534 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d21.nc 2024-07-20 03:03:43,374 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d15.nc 2024-07-20 03:03:43,542 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d16.nc 2024-07-20 03:03:43,712 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d17.nc 2024-07-20 03:03:43,911 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc 2024-07-20 03:03:44,091 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d19.nc 2024-07-20 03:03:44,370 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d20.nc 2024-07-20 03:03:44,566 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d21.nc 2024-07-20 03:03:44,769 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d22.nc 2024-07-20 03:03:44,840 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d22.nc 2024-07-20 03:03:44,846 INFO [grib_to_netcdf] 2024-07-20 NEMO-atmos forcing file for forecast2 created 2024-07-20 03:03:44,848 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker 2024-07-20 03:03:44,947 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d23.nc 2024-07-20 03:03:45,124 INFO [make_ssh_files] sea surface height boundary file for 2024-07-20 forecast2 run created 2024-07-20 03:03:45,126 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker 2024-07-20 03:03:45,352 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost') 2024-07-20 03:03:45,357 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost') 2024-07-20 03:03:46,503 INFO [make_runoff_file] running in process 696684 2024-07-20 03:03:46,504 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:46,505 INFO [make_runoff_file] publishing log messages to tcp://*:5562 2024-07-20 03:03:46,506 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:46,530 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry 2024-07-20 03:03:46,637 INFO [make_runoff_file] 2024-07-20 runoff file creation from Fraser at Hope and climatology elsewhere complete 2024-07-20 03:03:46,639 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker 2024-07-20 03:03:47,049 INFO [make_v202111_runoff_file] running in process 696682 2024-07-20 03:03:47,050 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:47,051 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5567 2024-07-20 03:03:47,052 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:47,053 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-19 2024-07-20 03:03:48,059 ERROR [make_v202111_runoff_file] no 2024-07-19 discharge obs for Skagit_MountVernon: patching 2024-07-20 03:03:48,515 ERROR [make_v202111_runoff_file] no 2024-07-19 discharge obs for Snohomish_Monroe: patching 2024-07-20 03:03:48,725 ERROR [make_v202111_runoff_file] no 2024-07-19 discharge obs for Nisqually_McKenna: patching 2024-07-20 03:03:48,819 ERROR [make_v202111_runoff_file] no 2024-07-19 discharge obs for Greenwater_Greenwater: patching 2024-07-20 03:03:50,099 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-19: /results/forcing/rivers/R202108Dailies_y2024m07d19.nc 2024-07-20 03:03:50,100 INFO [make_v202111_runoff_file] 2024-07-19 runoff file creation completed 2024-07-20 03:03:50,102 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker 2024-07-20 03:03:50,276 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost') 2024-07-20 03:03:50,281 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost') 2024-07-20 03:03:50,285 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost') 2024-07-20 03:03:50,289 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost') 2024-07-20 03:03:51,475 INFO [upload_forcing] running in process 696724 2024-07-20 03:03:51,477 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:51,478 INFO [upload_forcing] publishing log messages to tcp://*:5562 2024-07-20 03:03:51,478 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:51,496 INFO [upload_forcing] running in process 696725 2024-07-20 03:03:51,497 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:51,498 INFO [upload_forcing] publishing log messages to tcp://*:5567 2024-07-20 03:03:51,498 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:51,560 INFO [upload_forcing] running in process 696722 2024-07-20 03:03:51,561 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:51,562 INFO [upload_forcing] publishing log messages to tcp://*:5568 2024-07-20 03:03:51,563 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:51,616 INFO [upload_forcing] running in process 696723 2024-07-20 03:03:51,617 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:51,627 INFO [upload_forcing] publishing log messages to tcp://*:5569 2024-07-20 03:03:51,632 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:52,423 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-07-19 2024-07-20 03:03:52,866 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d19.nc 2024-07-20 03:03:52,964 INFO [upload_forcing] forecast2 2024-07-20 forcing files upload to optimum-hindcast completed 2024-07-20 03:03:52,968 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-20 03:03:53,166 INFO [upload_forcing] forecast2 2024-07-20 forcing files upload to orcinus-nowcast-agrif completed 2024-07-20 03:03:53,192 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-20 03:03:53,218 WARNING [get_onc_ferry] No ONC TWDP TSG temperature,conductivity,salinity data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:53,223 WARNING [get_onc_ferry] No ONC TWDP TSG temperature data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:53,225 WARNING [get_onc_ferry] No ONC TWDP TSG conductivity data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:53,226 WARNING [get_onc_ferry] No ONC TWDP TSG salinity data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:53,227 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-07-19 2024-07-20 03:03:53,689 WARNING [get_onc_ctd] no SEVIP salinity data 2024-07-20 03:03:53,973 INFO [get_onc_ctd] 2024-07-19 ONC SEVIP CTD T&S file created 2024-07-20 03:03:53,976 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker 2024-07-20 03:03:54,059 INFO [upload_forcing] forecast2 2024-07-20 forcing files upload to arbutus.cloud-nowcast completed 2024-07-20 03:03:54,135 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation,oxygen_corrected,temperature data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:54,141 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:54,143 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:54,145 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR temperature data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:54,146 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-07-19 2024-07-20 03:03:54,179 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost') 2024-07-20 03:03:54,184 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-20 03:03:54,383 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-20'], host='localhost') 2024-07-20 03:03:55,035 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence,chlorophyll,turbidity data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:55,041 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:55,042 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL chlorophyll data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:55,043 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL turbidity data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:55,044 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-07-19 2024-07-20 03:03:55,202 INFO [ping_erddap] running in process 696752 2024-07-20 03:03:55,203 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:55,204 INFO [ping_erddap] publishing log messages to tcp://*:5567 2024-07-20 03:03:55,205 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:55,207 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created 2024-07-20 03:03:55,209 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-20 03:03:55,447 INFO [make_forcing_links] running in process 696754 2024-07-20 03:03:55,449 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:55,450 INFO [make_forcing_links] publishing log messages to tcp://*:5568 2024-07-20 03:03:55,450 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:55,901 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:55,907 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:55,908 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2024-07-19; substituting empty dataset 2024-07-20 03:03:55,909 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-07-19 2024-07-20 03:03:56,616 INFO [make_forcing_links] forecast2 2024-07-20 forcing file links on arbutus.cloud-nowcast created 2024-07-20 03:03:56,621 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker 2024-07-20 03:03:56,825 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast') 2024-07-20 03:03:58,714 INFO [upload_forcing] forecast2 2024-07-20 forcing files upload to robot.graham completed 2024-07-20 03:03:58,718 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker 2024-07-20 03:03:58,971 INFO [run_NEMO] running in process 15237 2024-07-20 03:03:58,971 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:03:58,972 INFO [run_NEMO] publishing log messages to tcp://*:5556 2024-07-20 03:03:58,982 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:03:59,907 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/19jul24forecast2_2024-07-20T030359.052797-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast 2024-07-20 03:03:59,926 INFO [run_NEMO] forecast2 NEMO run for 2024-07-20 on arbutus.cloud-nowcast started 2024-07-20 03:03:59,930 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker 2024-07-20 03:04:00,133 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-07-20 03:04:02,152 INFO [watch_NEMO] running in process 15513 2024-07-20 03:04:02,152 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:04:02,153 INFO [watch_NEMO] publishing log messages to tcp://*:5557 2024-07-20 03:04:02,154 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:04:02,226 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch... 2024-07-20 03:04:13,732 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-07-19 2024-07-20 03:04:22,371 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-07-19 2024-07-20 03:04:30,545 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-07-19; substituting empty dataset 2024-07-20 03:04:30,548 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-07-19 2024-07-20 03:04:38,600 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-19; substituting empty dataset 2024-07-20 03:04:38,602 WARNING [get_onc_ferry] ONC TWDP TSG temperature data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,622 WARNING [get_onc_ferry] ONC TWDP TSG conductivity data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,684 WARNING [get_onc_ferry] ONC TWDP TSG salinity data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,702 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,719 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,736 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR temperature data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,753 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,770 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL chlorophyll data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,787 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL turbidity data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,804 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,822 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,922 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:04:38,939 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-19 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs 2024-07-20 03:05:53,478 INFO [get_onc_ferry] 2024-07-19 ONC TWDP ferry data file created 2024-07-20 03:05:53,481 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker 2024-07-20 03:05:53,665 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost') 2024-07-20 03:05:54,672 INFO [ping_erddap] running in process 696785 2024-07-20 03:05:54,673 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:05:54,674 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-07-20 03:05:54,675 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:05:54,677 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created 2024-07-20 03:05:54,678 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-20 03:09:02,230 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17798853 = 2024-07-21 05:01:20 UTC, 16.7% complete 2024-07-20 03:14:02,228 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17799332 = 2024-07-21 10:20:40 UTC, 34.5% complete 2024-07-20 03:19:02,229 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17799817 = 2024-07-21 15:44:00 UTC, 52.5% complete 2024-07-20 03:24:02,230 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17800300 = 2024-07-21 21:06:00 UTC, 70.4% complete 2024-07-20 03:29:02,230 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17800754 = 2024-07-22 02:08:40 UTC, 87.2% complete 2024-07-20 03:34:02,231 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17801100 = 2024-07-22 05:59:20 UTC, 100.0% complete 2024-07-20 03:39:02,235 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed 2024-07-20 03:39:02,239 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker 2024-07-20 03:39:02,627 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast') 2024-07-20 03:39:02,635 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast') 2024-07-20 03:39:02,639 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-19'], host='localhost') 2024-07-20 03:39:03,846 INFO [download_results] running in process 696960 2024-07-20 03:39:03,847 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:39:03,848 INFO [download_results] publishing log messages to tcp://*:5562 2024-07-20 03:39:03,849 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:39:03,850 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/19jul24 to /results/SalishSea/forecast2.202111 2024-07-20 03:39:05,494 INFO [make_ww3_current_file] running in process 16110 2024-07-20 03:39:05,495 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:39:05,496 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571 2024-07-20 03:39:05,497 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:39:05,498 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-20 forecast2 run 2024-07-20 03:39:05,703 INFO [make_ww3_wind_file] running in process 16108 2024-07-20 03:39:05,704 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:39:05,705 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570 2024-07-20 03:39:05,707 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:39:05,708 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-20 forecast2 run 2024-07-20 03:39:15,600 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-20 forecast2 run 2024-07-20 03:39:15,605 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker 2024-07-20 03:41:14,570 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-20 forecast2 run 2024-07-20 03:41:14,603 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker 2024-07-20 03:41:14,813 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-20'], host='arbutus.cloud-nowcast') 2024-07-20 03:41:24,192 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker 2024-07-20 03:41:24,400 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast') 2024-07-20 03:41:36,110 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch... 2024-07-20 03:46:36,041 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 99, 18.3% complete 2024-07-20 03:51:36,136 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete 2024-07-20 03:54:29,774 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker 2024-07-20 03:54:30,120 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-03-06'], host='localhost') 2024-07-20 03:54:30,127 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-20 03:54:30,132 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost') 2024-07-20 03:54:31,317 INFO [download_results] running in process 696996 2024-07-20 03:54:31,318 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 03:54:31,319 INFO [download_results] publishing log messages to tcp://*:5565 2024-07-20 03:54:31,320 INFO [download_results] connected to skookum.eos.ubc.ca port 5555 2024-07-20 03:54:31,321 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06mar18 to /ocean/sallen/timbucktwo/oxygen 2024-07-20 03:54:35,623 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker 2024-07-20 03:55:12,046 INFO [download_results] hindcast 2018-03-06 results files from optimum-hindcast downloaded 2024-07-20 03:55:12,057 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-20 03:55:12,264 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-03-06'], host='localhost') 2024-07-20 03:55:13,329 INFO [manager] checklist updated with [results splitting] items from split_results worker 2024-07-20 03:56:35,498 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 306, 56.7% complete 2024-07-20 04:01:35,847 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 408, 75.6% complete 2024-07-20 04:04:04,355 INFO [download_results] forecast2 2024-07-19 results files from arbutus.cloud-nowcast downloaded 2024-07-20 04:04:04,360 INFO [manager] checklist updated with [results files] items from download_results worker 2024-07-20 04:04:04,565 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-07-19'], host='localhost') 2024-07-20 04:04:06,388 INFO [make_CHS_currents_file] running in process 697063 2024-07-20 04:04:06,389 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:04:06,390 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562 2024-07-20 04:04:06,391 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:04:40,851 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-19 for forecast2 2024-07-20 04:04:40,855 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker 2024-07-20 04:04:41,060 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-07-19'], host='localhost') 2024-07-20 04:04:42,001 INFO [update_forecast_datasets] running in process 697088 2024-07-20 04:04:42,002 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:04:42,003 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562 2024-07-20 04:04:42,004 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:04:42,005 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/19jul24 run 2024-07-20 04:06:35,480 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 513, 95.0% complete 2024-07-20 04:08:53,597 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/19jul24 run: /results/SalishSea/rolling-forecasts/nemo 2024-07-20 04:08:53,598 INFO [update_forecast_datasets] nemo 2024-07-19 forecast2 rolling forecast datasets updated 2024-07-20 04:08:53,600 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-07-20 04:08:53,773 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost') 2024-07-20 04:08:53,778 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-07-19'], host='localhost') 2024-07-20 04:08:53,783 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-07-19'], host='localhost') 2024-07-20 04:08:54,985 INFO [ping_erddap] running in process 697164 2024-07-20 04:08:54,986 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:08:54,987 INFO [ping_erddap] publishing log messages to tcp://*:5562 2024-07-20 04:08:54,988 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:08:54,996 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created 2024-07-20 04:08:54,998 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-20 04:08:56,076 INFO [make_surface_current_tiles] running in process 697167 2024-07-20 04:08:56,077 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:08:56,078 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562 2024-07-20 04:08:56,079 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:08:56,109 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-19 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/19jul24 2024-07-20 04:08:59,420 INFO [make_plots] running in process 697166 2024-07-20 04:08:59,421 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:08:59,423 INFO [make_plots] publishing log messages to tcp://*:5565 2024-07-20 04:08:59,424 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:09:00,109 INFO [make_plots] preparing render list for 2024-07-19 NEMO forecast2 publish figures 2024-07-20 04:09:00,232 INFO [make_plots] added Website_thumbnail figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,233 INFO [make_plots] added Threshold_website figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,234 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,235 INFO [make_plots] added Boundary Bay figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,236 INFO [make_plots] added Campbell River figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,237 INFO [make_plots] added Cherry Point figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,237 INFO [make_plots] added Friday Harbor figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,238 INFO [make_plots] added Halfmoon Bay figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,239 INFO [make_plots] added Nanaimo figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,239 INFO [make_plots] added Neah Bay figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,240 INFO [make_plots] added New Westminster figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,240 INFO [make_plots] added Patricia Bay figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,241 INFO [make_plots] added Point Atkinson figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,242 INFO [make_plots] added Port Renfrew figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,242 INFO [make_plots] added Sand Heads figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,243 INFO [make_plots] added Sandy Cove figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,243 INFO [make_plots] added Squamish figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,244 INFO [make_plots] added Victoria figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,245 INFO [make_plots] added Woodwards Landing figure to 2024-07-19 NEMO forecast2 publish render list 2024-07-20 04:09:00,245 INFO [make_plots] starting to render nemo forecast2 publish 19jul24 figures 2024-07-20 04:11:35,482 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed 2024-07-20 04:11:35,484 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker 2024-07-20 04:11:36,048 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-20'], host='localhost') 2024-07-20 04:11:37,638 INFO [download_wwatch3_results] running in process 697741 2024-07-20 04:11:37,639 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:11:37,639 INFO [download_wwatch3_results] publishing log messages to tcp://*:5566 2024-07-20 04:11:37,640 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:13:54,831 INFO [download_wwatch3_results] forecast2 2024-07-20 results files from arbutus.cloud-nowcast downloaded 2024-07-20 04:13:54,834 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker 2024-07-20 04:13:55,243 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-07-20'], host='localhost') 2024-07-20 04:13:56,789 INFO [update_forecast_datasets] running in process 698123 2024-07-20 04:13:56,790 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:13:56,790 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566 2024-07-20 04:13:56,791 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:13:56,792 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/20jul24 run 2024-07-20 04:13:56,793 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/20jul24 run 2024-07-20 04:14:27,861 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/20jul24 run: /results/SalishSea/rolling-forecasts/wwatch3 2024-07-20 04:14:27,861 INFO [update_forecast_datasets] wwatch3 2024-07-20 forecast2 rolling forecast datasets updated 2024-07-20 04:14:27,865 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker 2024-07-20 04:14:28,058 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost') 2024-07-20 04:14:29,439 INFO [ping_erddap] running in process 698243 2024-07-20 04:14:29,440 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:14:29,441 INFO [ping_erddap] publishing log messages to tcp://*:5566 2024-07-20 04:14:29,441 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:14:29,443 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created 2024-07-20 04:14:29,445 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker 2024-07-20 04:14:29,850 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-07-20'], host='localhost') 2024-07-20 04:14:34,257 INFO [make_plots] running in process 698251 2024-07-20 04:14:34,258 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:14:34,259 INFO [make_plots] publishing log messages to tcp://*:5566 2024-07-20 04:14:34,260 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:14:34,260 INFO [make_plots] preparing render list for 2024-07-20 WaveWatch3 forecast2 publish figures 2024-07-20 04:14:34,261 INFO [make_plots] added Halibut Bank figure to 2024-07-20 WaveWatch3 forecast2 publish render list 2024-07-20 04:14:34,262 INFO [make_plots] added Sentry Shoal figure to 2024-07-20 WaveWatch3 forecast2 publish render list 2024-07-20 04:14:34,262 INFO [make_plots] starting to render wwatch3 forecast2 publish 20jul24 figures 2024-07-20 04:14:34,728 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank 2024-07-20 04:14:44,198 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal 2024-07-20 04:14:50,874 INFO [make_plots] finished rendering wwatch3 forecast2 publish 20jul24 figures 2024-07-20 04:14:50,874 INFO [make_plots] wwatch3 publish plots for 2024-07-20 forecast2 completed 2024-07-20 04:14:50,877 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-07-20 04:20:14,544 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/19jul24 2024-07-20 04:22:38,403 WARNING [make_plots] invalid station number for Boundary Bay station: None 2024-07-20 04:22:38,404 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay 2024-07-20 04:23:14,691 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:24:25,990 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/19jul24 2024-07-20 04:24:26,001 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-19 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/19jul24 2024-07-20 04:24:26,002 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-19 forecast2 completed 2024-07-20 04:24:26,310 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker 2024-07-20 04:25:04,181 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:25:04,523 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:25:39,817 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:26:50,694 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:27:26,896 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:28:02,890 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:28:39,346 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:29:14,219 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:29:50,327 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:30:26,932 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:30:27,299 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:31:02,843 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:31:37,887 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-15 00:00:00Z to 2024-07-22 05:55:00Z 2024-07-20 04:32:14,270 INFO [make_plots] finished rendering nemo forecast2 publish 19jul24 figures 2024-07-20 04:32:14,273 INFO [make_plots] nemo publish plots for 2024-07-19 forecast2 completed 2024-07-20 04:32:14,282 INFO [manager] checklist updated with [plots] items from make_plots worker 2024-07-20 04:32:14,604 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-07-19'], host='localhost') 2024-07-20 04:32:17,054 INFO [make_feeds] running in process 699285 2024-07-20 04:32:17,055 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:32:17,056 INFO [make_feeds] publishing log messages to tcp://*:5562 2024-07-20 04:32:17,058 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:32:45,516 INFO [make_feeds] ATOM feeds for 2024-07-19 forecast2 run completed 2024-07-20 04:32:45,519 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker 2024-07-20 04:32:45,854 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost') 2024-07-20 04:32:46,899 INFO [clear_checklist] running in process 699308 2024-07-20 04:32:46,900 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:32:46,901 INFO [clear_checklist] publishing log messages to tcp://*:5562 2024-07-20 04:32:46,902 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:32:46,903 INFO [clear_checklist] requesting that manager clear system state checklist 2024-07-20 04:32:46,904 INFO [manager] writing checklist to log file 2024-07-20 04:32:46,950 INFO [manager] checklist cleared 2024-07-20 04:32:46,953 INFO [clear_checklist] nowcast system checklist cleared 2024-07-20 04:32:46,955 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost') 2024-07-20 04:32:47,876 INFO [rotate_logs] running in process 699313 2024-07-20 04:32:47,877 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml 2024-07-20 04:32:47,878 INFO [rotate_logs] publishing log messages to tcp://*:5562 2024-07-20 04:32:47,879 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555 2024-07-20 04:32:47,880 INFO [rotate_logs] rotating log files