2024-07-18 04:31:07,139 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-07-18 04:31:07,145 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-07-18 04:31:07,149 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-07-18 04:31:07,150 INFO [rotate_logs] log files rotated
2024-07-18 04:31:07,154 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-07-18 04:58:54,623 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 04:58:54,788 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-06'], host='localhost')
2024-07-18 04:58:54,796 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 04:58:54,800 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 04:58:56,071 INFO [download_results] running in process 658479
2024-07-18 04:58:56,072 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 04:58:56,073 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-18 04:58:56,075 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 04:58:56,076 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 04:59:00,270 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 04:59:35,961 INFO [download_results] hindcast 2018-10-06 results files from optimum-hindcast downloaded
2024-07-18 04:59:35,972 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 04:59:35,979 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-06'], host='localhost')
2024-07-18 04:59:37,125 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 06:24:04,789 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 06:24:04,954 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-11'], host='localhost')
2024-07-18 06:24:04,961 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 06:24:04,966 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 06:24:06,178 INFO [download_results] running in process 659860
2024-07-18 06:24:06,179 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 06:24:06,181 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-18 06:24:06,182 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 06:24:06,183 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 06:24:10,258 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 06:24:45,864 INFO [download_results] hindcast 2018-10-11 results files from optimum-hindcast downloaded
2024-07-18 06:24:45,875 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 06:24:45,881 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-11'], host='localhost')
2024-07-18 06:24:47,051 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 07:49:14,739 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 07:49:14,991 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-16'], host='localhost')
2024-07-18 07:49:14,998 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 07:49:15,004 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 07:49:16,175 INFO [download_results] running in process 661131
2024-07-18 07:49:16,176 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 07:49:16,177 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-18 07:49:16,178 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 07:49:16,179 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 07:49:20,450 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 07:49:55,841 INFO [download_results] hindcast 2018-10-16 results files from optimum-hindcast downloaded
2024-07-18 07:49:55,850 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 07:49:55,858 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-16'], host='localhost')
2024-07-18 07:49:56,919 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 09:12:54,301 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240718/12/
2024-07-18 09:12:54,302 INFO [collect_weather] 2024-07-18 2.5km weather forecast 12 collection complete
2024-07-18 09:12:54,305 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-18 09:12:54,468 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-07-17'], host='localhost')
2024-07-18 09:12:54,476 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-07-17'], host='localhost')
2024-07-18 09:12:54,480 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-07-17'], host='localhost')
2024-07-18 09:12:54,485 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-07-17'], host='localhost')
2024-07-18 09:12:54,490 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-07-18 09:12:54,496 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-07-18 09:12:54,502 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-07-18 09:12:54,509 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-07-18 09:12:54,517 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-07-18 09:12:54,536 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240718/12/
2024-07-18 09:12:54,537 INFO [crop_gribs] 2024-07-18 12 GRIBs cropping complete
2024-07-18 09:12:54,540 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-18 09:12:54,695 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-07-18 09:12:56,248 INFO [collect_NeahBay_ssh] running in process 662846
2024-07-18 09:12:56,249 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:56,251 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2024-07-18 09:12:56,252 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:56,254 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-07-18
2024-07-18 09:12:56,294 INFO [collect_weather] running in process 662849
2024-07-18 09:12:56,295 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:56,298 INFO [collect_weather] publishing log messages to tcp://*:5565
2024-07-18 09:12:56,301 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:56,315 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-07-18 09:12:56,629 INFO [download_live_ocean] running in process 662848
2024-07-18 09:12:56,630 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:56,631 INFO [download_live_ocean] publishing log messages to tcp://*:5566
2024-07-18 09:12:56,632 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:56,632 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-07-18
2024-07-18 09:12:56,923 INFO [collect_river_data] running in process 662843
2024-07-18 09:12:56,924 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:56,924 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-18 09:12:56,925 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:56,926 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-07-17
2024-07-18 09:12:56,927 INFO [collect_river_data] running in process 662842
2024-07-18 09:12:56,928 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:56,929 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-18 09:12:56,930 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:56,930 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-07-17
2024-07-18 09:12:56,932 INFO [collect_river_data] running in process 662844
2024-07-18 09:12:56,933 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:56,934 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-18 09:12:56,935 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:56,936 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-07-17
2024-07-18 09:12:56,998 INFO [make_turbidity_file] running in process 662845
2024-07-18 09:12:57,000 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:57,001 INFO [make_turbidity_file] publishing log messages to tcp://*:5570
2024-07-18 09:12:57,003 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:57,004 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-07-18
2024-07-18 09:12:57,015 INFO [collect_river_data] running in process 662840
2024-07-18 09:12:57,019 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:57,020 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-18 09:12:57,023 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:57,037 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-07-17
2024-07-18 09:12:57,178 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-07-17&endDT=2024-07-17 "HTTP/1.1 200 OK"
2024-07-18 09:12:57,183 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-07-17 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-07-18 09:12:57,184 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-07-17 completed
2024-07-18 09:12:57,187 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-18 09:12:57,198 INFO [grib_to_netcdf] running in process 662862
2024-07-18 09:12:57,200 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:57,201 INFO [grib_to_netcdf] publishing log messages to tcp://*:5572
2024-07-18 09:12:57,203 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:57,205 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-18 nowcast and forecast runs
2024-07-18 09:12:57,262 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-07-17&endDT=2024-07-17 "HTTP/1.1 200 OK"
2024-07-18 09:12:57,262 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-07-17&endDT=2024-07-17 "HTTP/1.1 200 OK"
2024-07-18 09:12:57,272 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-07-17 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-07-18 09:12:57,272 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-07-17 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-07-18 09:12:57,273 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-07-17 completed
2024-07-18 09:12:57,274 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-07-17 completed
2024-07-18 09:12:57,276 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-07-17&endDT=2024-07-17 "HTTP/1.1 200 OK"
2024-07-18 09:12:57,278 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-18 09:12:57,280 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-18 09:12:57,282 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-07-17 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-07-18 09:12:57,283 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-07-17 completed
2024-07-18 09:12:57,287 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-18 09:12:57,356 INFO [crop_gribs] running in process 662850
2024-07-18 09:12:57,357 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:12:57,358 INFO [crop_gribs] publishing log messages to tcp://*:5573
2024-07-18 09:12:57,358 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:12:57,359 INFO [crop_gribs] cropping 2024-07-18 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-07-18 09:12:57,373 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240718/18/
2024-07-18 09:12:57,439 INFO [download_live_ocean] downloaded 10152219 bytes from https://liveocean.apl.uw.edu/output/f2024.07.18/ubc.nc to /results/forcing/LiveOcean/downloaded/20240718/low_passed_UBC.nc
2024-07-18 09:12:57,440 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-07-18 09:12:57,699 INFO [collect_NeahBay_ssh] 2024-07-18T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-07-18 09:12:57,702 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-18 09:12:57,710 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 09:12:58,485 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240718/low_passed_UBC.nc
2024-07-18 09:12:58,486 INFO [download_live_ocean] 2024-07-18 Live Ocean file for Salish Sea western boundary downloaded
2024-07-18 09:12:58,488 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-07-18 09:12:58,657 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-07-18'], host='localhost')
2024-07-18 09:13:00,084 INFO [make_turbidity_file] 2024-07-18 Fraser River turbidity file creation complete
2024-07-18 09:13:00,087 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-07-18 09:13:00,422 INFO [make_ssh_files] running in process 663022
2024-07-18 09:13:00,423 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:00,425 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2024-07-18 09:13:00,426 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:00,428 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-07-18 NOAA Neah Bay 06Z observation and forecast values
2024-07-18 09:13:00,782 INFO [make_live_ocean_files] running in process 663035
2024-07-18 09:13:00,783 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:00,784 INFO [make_live_ocean_files] publishing log messages to tcp://*:5566
2024-07-18 09:13:00,786 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:00,789 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-07-18 Live Ocean run
2024-07-18 09:13:04,821 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d14.nc
2024-07-18 09:13:05,003 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d15.nc
2024-07-18 09:13:05,188 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d16.nc
2024-07-18 09:13:05,368 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d17.nc
2024-07-18 09:13:05,563 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d18.nc
2024-07-18 09:13:05,745 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d19.nc
2024-07-18 09:13:05,927 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d20.nc
2024-07-18 09:13:06,108 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d21.nc
2024-07-18 09:13:06,221 INFO [make_ssh_files] sea surface height boundary file for 2024-07-18 nowcast run created
2024-07-18 09:13:06,223 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-18 09:13:06,230 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-18 09:13:06,234 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-18 09:13:06,276 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m07d18.nc
2024-07-18 09:13:07,587 INFO [make_runoff_file] running in process 663091
2024-07-18 09:13:07,588 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:07,590 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-07-18 09:13:07,591 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:07,616 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-18 09:13:07,703 INFO [make_runoff_file] 2024-07-18 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-18 09:13:07,704 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-18 09:13:08,154 INFO [make_v202111_runoff_file] running in process 663089
2024-07-18 09:13:08,155 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:08,156 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564
2024-07-18 09:13:08,157 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:08,158 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-17
2024-07-18 09:13:10,631 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-17: /results/forcing/rivers/R202108Dailies_y2024m07d17.nc
2024-07-18 09:13:10,632 INFO [make_v202111_runoff_file] 2024-07-17 runoff file creation completed
2024-07-18 09:13:10,634 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-18 09:13:13,368 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d19.nc
2024-07-18 09:13:17,265 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d20.nc
2024-07-18 09:13:17,279 INFO [grib_to_netcdf] 2024-07-18 NEMO-atmos forcing file for nowcast+ created
2024-07-18 09:13:17,281 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-18 09:13:40,189 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d18.nc
2024-07-18 09:13:40,190 INFO [make_live_ocean_files] 2024-07-18 Live Ocean western boundary conditions files created
2024-07-18 09:13:40,193 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-07-18 09:13:40,200 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-07-18 09:13:40,204 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-07-18 09:13:40,207 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-07-18 09:13:40,212 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-07-18 09:13:40,217 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-07-18 09:13:41,380 INFO [ping_erddap] running in process 663131
2024-07-18 09:13:41,381 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:41,382 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-18 09:13:41,383 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:41,385 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-07-18 09:13:41,387 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-18 09:13:41,510 INFO [upload_forcing] running in process 663136
2024-07-18 09:13:41,511 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:41,513 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-07-18 09:13:41,514 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:41,527 INFO [upload_forcing] running in process 663135
2024-07-18 09:13:41,529 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:41,530 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-18 09:13:41,531 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:41,533 INFO [upload_forcing] running in process 663133
2024-07-18 09:13:41,535 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:41,536 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-18 09:13:41,536 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:41,548 INFO [upload_forcing] running in process 663134
2024-07-18 09:13:41,549 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:41,550 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-18 09:13:41,551 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:43,823 INFO [upload_forcing] nowcast+ 2024-07-18 forcing files upload to optimum-hindcast completed
2024-07-18 09:13:43,827 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-18 09:13:43,964 INFO [upload_forcing] nowcast+ 2024-07-18 forcing files upload to orcinus-nowcast-agrif completed
2024-07-18 09:13:43,967 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-18 09:13:51,899 INFO [upload_forcing] nowcast+ 2024-07-18 forcing files upload to robot.graham completed
2024-07-18 09:13:51,904 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-18 09:13:53,525 INFO [upload_forcing] nowcast+ 2024-07-18 forcing files upload to arbutus.cloud-nowcast completed
2024-07-18 09:13:53,530 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-18 09:13:53,543 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 09:13:54,654 INFO [make_forcing_links] running in process 663170
2024-07-18 09:13:54,655 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:54,657 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-18 09:13:54,658 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:55,806 INFO [make_forcing_links] nowcast+ 2024-07-18 forcing file links on arbutus.cloud-nowcast created
2024-07-18 09:13:55,810 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-18 09:13:55,822 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-18'], host='arbutus.cloud-nowcast')
2024-07-18 09:13:57,964 INFO [run_NEMO] running in process 15324
2024-07-18 09:13:57,966 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:13:57,967 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-18 09:13:58,153 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:13:58,870 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/18jul24nowcast_2024-07-18T091358.014307-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-18 09:13:58,891 INFO [run_NEMO] nowcast NEMO run for 2024-07-18 on arbutus.cloud-nowcast started
2024-07-18 09:13:58,896 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-18 09:13:58,908 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-18 09:14:00,937 INFO [watch_NEMO] running in process 15580
2024-07-18 09:14:00,938 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:14:00,939 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-18 09:14:00,941 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:14:01,009 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-18 09:14:24,784 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 09:14:24,953 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-21'], host='localhost')
2024-07-18 09:14:24,962 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 09:14:24,966 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 09:14:26,371 INFO [download_results] running in process 663181
2024-07-18 09:14:26,372 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:14:26,373 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-18 09:14:26,375 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:14:26,376 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 09:14:30,462 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 09:15:06,629 INFO [download_results] hindcast 2018-10-21 results files from optimum-hindcast downloaded
2024-07-18 09:15:06,639 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 09:15:06,653 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-21'], host='localhost')
2024-07-18 09:15:30,221 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 09:19:01,010 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17792341 = 2024-07-18 04:40:00 UTC, 19.5% complete
2024-07-18 09:24:01,011 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17792822 = 2024-07-18 10:00:40 UTC, 41.7% complete
2024-07-18 09:29:01,013 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17793284 = 2024-07-18 15:08:40 UTC, 63.1% complete
2024-07-18 09:34:01,014 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17793748 = 2024-07-18 20:18:00 UTC, 84.6% complete
2024-07-18 09:39:01,015 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17794080 = 2024-07-18 23:59:20 UTC, 100.0% complete
2024-07-18 09:44:01,021 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-07-18 09:44:01,024 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-18 09:44:01,206 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 09:44:01,212 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 09:44:02,832 INFO [make_forcing_links] running in process 663589
2024-07-18 09:44:02,833 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:44:02,834 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-07-18 09:44:02,835 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:44:02,837 INFO [download_results] running in process 663590
2024-07-18 09:44:02,838 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:44:02,840 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-18 09:44:02,842 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:44:02,843 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/18jul24 to /results/SalishSea/nowcast-blue.202111
2024-07-18 09:44:03,763 INFO [make_forcing_links] ssh 2024-07-18 forcing file links on arbutus.cloud-nowcast created
2024-07-18 09:44:03,767 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-18 09:44:03,781 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-18'], host='arbutus.cloud-nowcast')
2024-07-18 09:44:06,003 INFO [run_NEMO] running in process 18502
2024-07-18 09:44:06,004 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:44:06,005 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-18 09:44:06,006 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:44:07,007 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/18jul24forecast_2024-07-18T094406.075066-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-18 09:44:07,027 INFO [run_NEMO] forecast NEMO run for 2024-07-18 on arbutus.cloud-nowcast started
2024-07-18 09:44:07,031 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-18 09:44:07,046 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-18 09:44:09,579 INFO [watch_NEMO] running in process 18763
2024-07-18 09:44:09,581 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 09:44:09,582 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-18 09:44:09,583 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-18 09:44:09,917 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-18 09:49:09,971 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17794526 = 2024-07-19 04:56:40 UTC, 13.7% complete
2024-07-18 09:54:09,930 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17794994 = 2024-07-19 10:08:40 UTC, 28.2% complete
2024-07-18 09:59:10,350 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17795470 = 2024-07-19 15:26:00 UTC, 42.9% complete
2024-07-18 10:04:09,935 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17795933 = 2024-07-19 20:34:40 UTC, 57.2% complete
2024-07-18 10:04:18,038 INFO [download_results] nowcast 2024-07-18 results files from arbutus.cloud-nowcast downloaded
2024-07-18 10:04:18,043 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 10:04:18,058 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:04:18,063 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-07-17'], host='localhost')
2024-07-18 10:04:18,067 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:04:19,923 INFO [make_CHS_currents_file] running in process 663753
2024-07-18 10:04:19,924 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:04:19,925 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-07-18 10:04:19,926 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:04:21,390 INFO [make_plots] running in process 663752
2024-07-18 10:04:21,391 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:04:21,392 INFO [make_plots] publishing log messages to tcp://*:5564
2024-07-18 10:04:21,393 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:04:21,497 INFO [make_plots] running in process 663750
2024-07-18 10:04:21,514 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:04:21,522 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-18 10:04:21,528 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:04:22,513 INFO [make_plots] preparing render list for 2024-07-17 NEMO nowcast-blue comparison figures
2024-07-18 10:04:22,514 INFO [make_plots] preparing render list for 2024-07-18 NEMO nowcast-blue research figures
2024-07-18 10:04:23,569 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-blue research render list
2024-07-18 10:04:24,238 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-blue research render list
2024-07-18 10:04:24,239 INFO [make_plots] added Currents_sections_and_surface figure to 2024-07-18 NEMO nowcast-blue research render list
2024-07-18 10:04:24,239 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-07-18 NEMO nowcast-blue research render list
2024-07-18 10:04:24,240 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-07-18 NEMO nowcast-blue research render list
2024-07-18 10:04:24,241 INFO [make_plots] starting to render nemo nowcast research 18jul24 figures
2024-07-18 10:04:49,375 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-18 for nowcast
2024-07-18 10:04:49,378 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-18 10:05:04,354 INFO [make_plots] added SH_wind figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,355 INFO [make_plots] added Compare_VENUS_East figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,356 INFO [make_plots] added Compare_VENUS_Central figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,357 INFO [make_plots] added Central_ADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,358 INFO [make_plots] added Central_depavADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,358 INFO [make_plots] added Central_timeavADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,359 INFO [make_plots] added East_ADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,360 INFO [make_plots] added East_depavADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,361 INFO [make_plots] added East_timeavADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,362 INFO [make_plots] added ddl_ADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,363 INFO [make_plots] added ddl_depavADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,364 INFO [make_plots] added ddl_timeavADCP figure to 2024-07-17 NEMO nowcast-blue comparison render list
2024-07-18 10:05:04,365 INFO [make_plots] starting to render nemo nowcast comparison 17jul24 figures
2024-07-18 10:05:33,552 INFO [make_plots] finished rendering nemo nowcast research 18jul24 figures
2024-07-18 10:05:33,555 INFO [make_plots] nemo research plots for 2024-07-18 nowcast completed
2024-07-18 10:05:33,570 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-18 10:06:22,195 WARNING [make_plots] VENUS Central node CTD comparison figure failed: No observations available
2024-07-18 10:06:22,704 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-18 10:06:23,206 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-18 10:06:23,701 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-07-18 10:06:24,194 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-18 10:06:24,686 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-18 10:06:25,179 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-07-18 10:06:25,673 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-18 10:06:26,166 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-18 10:06:26,659 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-07-18 10:06:26,660 INFO [make_plots] finished rendering nemo nowcast comparison 17jul24 figures
2024-07-18 10:06:26,746 INFO [make_plots] nemo comparison plots for 2024-07-17 nowcast completed
2024-07-18 10:06:26,749 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-18 10:09:09,873 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17796411 = 2024-07-20 01:53:20 UTC, 71.9% complete
2024-07-18 10:14:09,870 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17796887 = 2024-07-20 07:10:40 UTC, 86.6% complete
2024-07-18 10:19:09,880 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17797320 = 2024-07-20 11:59:20 UTC, 100.0% complete
2024-07-18 10:24:09,881 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-18 10:24:09,885 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-07-18 10:24:10,051 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-18'], host='arbutus.cloud-nowcast')
2024-07-18 10:24:10,059 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-18'], host='arbutus.cloud-nowcast')
2024-07-18 10:24:10,063 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:24:10,069 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:24:10,073 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:24:10,079 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:24:10,083 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:24:11,286 INFO [upload_forcing] running in process 664108
2024-07-18 10:24:11,287 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:11,288 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-07-18 10:24:11,289 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:11,290 INFO [upload_forcing] running in process 664111
2024-07-18 10:24:11,291 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:11,292 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-07-18 10:24:11,293 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:11,298 INFO [upload_forcing] running in process 664109
2024-07-18 10:24:11,299 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:11,300 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-18 10:24:11,301 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:11,304 INFO [upload_forcing] running in process 664110
2024-07-18 10:24:11,305 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:11,306 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-07-18 10:24:11,307 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:11,326 INFO [download_results] running in process 664112
2024-07-18 10:24:11,327 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:11,328 INFO [download_results] publishing log messages to tcp://*:5569
2024-07-18 10:24:11,329 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:11,330 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/18jul24 to /results/SalishSea/forecast.202111
2024-07-18 10:24:11,910 INFO [upload_forcing] turbidity 2024-07-18 forcing files upload to optimum-hindcast completed
2024-07-18 10:24:11,915 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-18 10:24:12,112 INFO [upload_forcing] turbidity 2024-07-18 forcing files upload to orcinus-nowcast-agrif completed
2024-07-18 10:24:12,116 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-18 10:24:12,134 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:24:12,295 INFO [upload_forcing] turbidity 2024-07-18 forcing files upload to arbutus.cloud-nowcast completed
2024-07-18 10:24:12,299 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-18 10:24:12,318 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:24:12,870 INFO [make_ww3_wind_file] running in process 21961
2024-07-18 10:24:12,871 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:12,872 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-18 10:24:12,882 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:12,883 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-18 forecast run
2024-07-18 10:24:12,948 INFO [make_ww3_current_file] running in process 21963
2024-07-18 10:24:12,949 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:12,950 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-18 10:24:12,951 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:12,951 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-18 forecast run
2024-07-18 10:24:13,270 INFO [make_forcing_links] running in process 664145
2024-07-18 10:24:13,272 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:13,273 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-07-18 10:24:13,274 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:13,582 INFO [make_forcing_links] running in process 664147
2024-07-18 10:24:13,583 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:13,584 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-07-18 10:24:13,584 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:14,111 INFO [make_forcing_links] nowcast-agrif 2024-07-18 forcing file links on orcinus-nowcast-agrif created
2024-07-18 10:24:14,115 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-18 10:24:14,132 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:24:15,236 INFO [upload_forcing] turbidity 2024-07-18 forcing files upload to robot.graham completed
2024-07-18 10:24:15,241 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-18 10:24:15,265 INFO [run_NEMO_agrif] running in process 664161
2024-07-18 10:24:15,267 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:15,268 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5562
2024-07-18 10:24:15,268 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:17,784 INFO [make_forcing_links] nowcast-green 2024-07-18 forcing file links on arbutus.cloud-nowcast created
2024-07-18 10:24:17,789 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-18 10:24:17,808 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-18'], host='arbutus.cloud-nowcast')
2024-07-18 10:24:23,225 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/18jul24nowcast-green_2024-07-18T102422.199863-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-18 10:24:23,247 INFO [run_NEMO] nowcast-green NEMO run for 2024-07-18 on arbutus.cloud-nowcast started
2024-07-18 10:24:23,254 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-18 10:24:23,272 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-07-18 10:24:23,932 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-18 forecast run
2024-07-18 10:24:23,936 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-18 10:24:27,183 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-18 10:24:27,783 INFO [run_NEMO_agrif] job id for 18jul24nowcast-agrif: 11101958.orca2.ibb
2024-07-18 10:24:27,785 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-07-18 queued on orcinus-nowcast-agrif
2024-07-18 10:24:27,789 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-07-18 10:24:27,807 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11101958.orca2.ibb'], host='localhost')
2024-07-18 10:24:28,933 INFO [watch_NEMO_agrif] running in process 664180
2024-07-18 10:24:28,934 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:24:28,935 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5562
2024-07-18 10:24:28,936 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:24:29,728 INFO [watch_NEMO_agrif] watching 18jul24nowcast-agrif job 11101958 on orcinus-nowcast-agrif
2024-07-18 10:24:29,959 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif job 11101958 is queued on orcinus-nowcast-agrif
2024-07-18 10:25:13,516 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-18 forecast run
2024-07-18 10:25:13,556 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-18 10:25:13,577 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-18'], host='arbutus.cloud-nowcast')
2024-07-18 10:25:24,754 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-18 10:25:24,775 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-07-18 10:25:30,866 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-07-18 10:25:36,606 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-18 10:29:27,685 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17792081 = 2024-07-18 01:46:40 UTC, 7.4% complete
2024-07-18 10:30:31,336 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7812935 = 2024-07-18 02:22:40 UTC, 9.9% complete
2024-07-18 10:30:36,611 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 75, 17.4% complete
2024-07-18 10:34:27,721 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17792258 = 2024-07-18 03:44:40 UTC, 15.6% complete
2024-07-18 10:35:31,810 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7813179 = 2024-07-18 05:05:20 UTC, 21.2% complete
2024-07-18 10:35:36,633 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 156, 36.1% complete
2024-07-18 10:39:27,699 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17792437 = 2024-07-18 05:44:00 UTC, 23.9% complete
2024-07-18 10:40:32,250 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7813428 = 2024-07-18 07:51:20 UTC, 32.7% complete
2024-07-18 10:40:36,634 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 237, 54.9% complete
2024-07-18 10:44:27,687 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17792615 = 2024-07-18 07:42:40 UTC, 32.1% complete
2024-07-18 10:45:32,692 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7813676 = 2024-07-18 10:36:40 UTC, 44.2% complete
2024-07-18 10:45:36,601 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 315, 72.9% complete
2024-07-18 10:49:27,687 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17792794 = 2024-07-18 09:42:00 UTC, 40.4% complete
2024-07-18 10:50:33,175 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7813922 = 2024-07-18 13:20:40 UTC, 55.6% complete
2024-07-18 10:50:36,588 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 396, 91.7% complete
2024-07-18 10:53:53,181 INFO [download_results] forecast 2024-07-18 results files from arbutus.cloud-nowcast downloaded
2024-07-18 10:53:53,185 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 10:53:53,203 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:53:54,969 INFO [make_CHS_currents_file] running in process 664533
2024-07-18 10:53:54,970 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:53:54,971 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-07-18 10:53:54,972 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:54:27,026 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17792973 = 2024-07-18 11:41:20 UTC, 48.7% complete
2024-07-18 10:54:34,889 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-18 for forecast
2024-07-18 10:54:34,893 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-18 10:54:34,914 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:54:35,947 INFO [update_forecast_datasets] running in process 664564
2024-07-18 10:54:35,948 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:54:35,949 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2024-07-18 10:54:35,950 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:54:35,951 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/18jul24 run
2024-07-18 10:54:36,238 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/18jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-18 10:54:36,238 INFO [update_forecast_datasets] nemo 2024-07-18 forecast rolling forecast datasets updated
2024-07-18 10:54:36,240 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-18 10:54:36,259 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-18 10:54:36,263 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:54:36,267 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:54:37,305 INFO [ping_erddap] running in process 664569
2024-07-18 10:54:37,306 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:54:37,307 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-18 10:54:37,308 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:54:37,317 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-18 10:54:37,318 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-18 10:54:38,348 INFO [make_surface_current_tiles] running in process 664572
2024-07-18 10:54:38,350 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:54:38,350 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2024-07-18 10:54:38,351 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:54:38,353 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-18 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/18jul24
2024-07-18 10:54:40,018 INFO [make_plots] running in process 664571
2024-07-18 10:54:40,018 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:54:40,019 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-18 10:54:40,020 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:54:40,723 INFO [make_plots] preparing render list for 2024-07-18 NEMO forecast publish figures
2024-07-18 10:54:40,845 INFO [make_plots] added Website_thumbnail figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,846 INFO [make_plots] added Threshold_website figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,847 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,848 INFO [make_plots] added Boundary Bay figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,849 INFO [make_plots] added Campbell River figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,849 INFO [make_plots] added Cherry Point figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,850 INFO [make_plots] added Friday Harbor figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,851 INFO [make_plots] added Halfmoon Bay figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,851 INFO [make_plots] added Nanaimo figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,852 INFO [make_plots] added Neah Bay figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,853 INFO [make_plots] added New Westminster figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,854 INFO [make_plots] added Patricia Bay figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,854 INFO [make_plots] added Point Atkinson figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,855 INFO [make_plots] added Port Renfrew figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,856 INFO [make_plots] added Sand Heads figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,857 INFO [make_plots] added Sandy Cove figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,857 INFO [make_plots] added Squamish figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,858 INFO [make_plots] added Victoria figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,859 INFO [make_plots] added Woodwards Landing figure to 2024-07-18 NEMO forecast publish render list
2024-07-18 10:54:40,859 INFO [make_plots] starting to render nemo forecast publish 18jul24 figures
2024-07-18 10:55:33,630 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7814169 = 2024-07-18 16:05:20 UTC, 67.1% complete
2024-07-18 10:55:35,939 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-18 10:55:35,942 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-18 10:55:36,151 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 10:55:36,156 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-18'], host='arbutus.cloud-nowcast')
2024-07-18 10:55:37,477 INFO [download_wwatch3_results] running in process 664930
2024-07-18 10:55:37,478 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:55:37,478 INFO [download_wwatch3_results] publishing log messages to tcp://*:5568
2024-07-18 10:55:37,479 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:55:38,179 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-18 10:55:38,211 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-07-18 10:55:40,239 INFO [watch_ww3] running in process 25811
2024-07-18 10:55:40,240 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:55:40,241 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-07-18 10:55:40,241 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:55:40,275 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-18 10:57:39,412 INFO [download_wwatch3_results] nowcast 2024-07-18 results files from arbutus.cloud-nowcast downloaded
2024-07-18 10:57:39,416 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-18 10:59:27,034 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17793149 = 2024-07-18 13:38:40 UTC, 56.9% complete
2024-07-18 10:59:37,072 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 10:59:37,257 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-10-26'], host='localhost')
2024-07-18 10:59:37,262 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 10:59:37,266 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 10:59:38,957 INFO [download_results] running in process 665603
2024-07-18 10:59:38,958 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 10:59:38,958 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-18 10:59:38,959 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 10:59:38,960 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26oct18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 10:59:43,335 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 11:00:29,957 INFO [download_results] hindcast 2018-10-26 results files from optimum-hindcast downloaded
2024-07-18 11:00:29,969 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 11:00:29,987 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-10-26'], host='localhost')
2024-07-18 11:00:31,111 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 11:00:31,144 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-oct', 'robot.graham'], host='localhost')
2024-07-18 11:00:32,166 INFO [archive_tarball] running in process 665769
2024-07-18 11:00:32,167 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:00:32,168 INFO [archive_tarball] publishing log messages to tcp://*:5566
2024-07-18 11:00:32,169 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:00:32,170 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct18.tar from /ocean/sallen/timbucktwo/oxygen/*oct18/
2024-07-18 11:00:34,102 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7814419 = 2024-07-18 18:52:00 UTC, 78.6% complete
2024-07-18 11:00:40,276 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 72, 11.1% complete
2024-07-18 11:04:27,032 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17793328 = 2024-07-18 15:38:00 UTC, 65.2% complete
2024-07-18 11:05:20,416 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/18jul24
2024-07-18 11:05:34,540 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7814666 = 2024-07-18 21:36:40 UTC, 90.1% complete
2024-07-18 11:05:40,277 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 153, 23.6% complete
2024-07-18 11:06:21,438 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-oct18.index from /ocean/dlatorne/oxygen-oct18.tar
2024-07-18 11:06:21,513 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-oct18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-18 11:08:09,936 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-18 11:08:09,937 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-18 11:08:47,256 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:09:27,029 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17793505 = 2024-07-18 17:36:00 UTC, 73.4% complete
2024-07-18 11:10:12,895 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/18jul24
2024-07-18 11:10:12,907 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-18 forecast into /results/nowcast-sys/figures/surface_currents/forecast/18jul24
2024-07-18 11:10:12,908 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-18 forecast completed
2024-07-18 11:10:13,226 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-18 11:10:35,017 INFO [watch_NEMO_agrif] 18jul24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7814880 = 2024-07-18 23:59:20 UTC, 100.0% complete
2024-07-18 11:10:40,278 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 234, 36.1% complete
2024-07-18 11:10:41,765 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:10:42,113 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:11:18,877 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:12:39,030 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:13:16,712 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:13:54,395 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:14:27,039 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17793684 = 2024-07-18 19:35:20 UTC, 81.7% complete
2024-07-18 11:14:31,584 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:15:08,371 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:15:35,233 INFO [watch_NEMO_agrif] NEMO AGRIF run 11101958.orca2.ibb on orcinus-nowcast-agrif completed
2024-07-18 11:15:35,237 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-07-18 11:15:35,554 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 11:15:37,032 INFO [download_results] running in process 666702
2024-07-18 11:15:37,033 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:15:37,034 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-18 11:15:37,035 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:15:37,036 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/18jul24 to /results/SalishSea/nowcast-agrif.201702
2024-07-18 11:15:40,279 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete
2024-07-18 11:15:46,960 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:16:00,703 INFO [download_results] nowcast-agrif 2024-07-18 results files from orcinus-nowcast-agrif downloaded
2024-07-18 11:16:00,709 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 11:16:00,877 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 11:16:04,498 INFO [make_plots] running in process 666712
2024-07-18 11:16:04,500 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:16:04,501 INFO [make_plots] publishing log messages to tcp://*:5562
2024-07-18 11:16:04,502 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:16:04,960 INFO [make_plots] preparing render list for 2024-07-18 NEMO nowcast-agrif research figures
2024-07-18 11:16:04,961 INFO [make_plots] added baynes_sound_surface figure to 2024-07-18 NEMO nowcast-agrif research render list
2024-07-18 11:16:04,962 INFO [make_plots] starting to render nemo nowcast-agrif research 18jul24 figures
2024-07-18 11:16:14,097 INFO [make_plots] finished rendering nemo nowcast-agrif research 18jul24 figures
2024-07-18 11:16:14,101 INFO [make_plots] nemo research plots for 2024-07-18 nowcast-agrif completed
2024-07-18 11:16:14,104 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-18 11:16:26,375 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:16:26,736 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:16:35,639 INFO [archive_tarball] hindcast *oct18 results files archived to robot.graham
2024-07-18 11:16:35,643 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-18 11:17:01,457 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:17:36,495 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-13 00:00:00Z to 2024-07-20 11:55:00Z
2024-07-18 11:18:11,789 INFO [make_plots] finished rendering nemo forecast publish 18jul24 figures
2024-07-18 11:18:11,791 INFO [make_plots] nemo publish plots for 2024-07-18 forecast completed
2024-07-18 11:18:11,798 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-18 11:18:11,976 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 11:18:17,410 INFO [make_feeds] running in process 666787
2024-07-18 11:18:17,413 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:18:17,414 INFO [make_feeds] publishing log messages to tcp://*:5562
2024-07-18 11:18:17,416 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:18:45,505 INFO [make_feeds] ATOM feeds for 2024-07-18 forecast run completed
2024-07-18 11:18:45,509 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-18 11:19:27,035 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17793862 = 2024-07-18 21:34:00 UTC, 89.9% complete
2024-07-18 11:20:40,280 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 390, 60.2% complete
2024-07-18 11:24:27,037 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17794039 = 2024-07-18 23:32:00 UTC, 98.1% complete
2024-07-18 11:25:40,281 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 471, 72.7% complete
2024-07-18 11:29:27,034 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17794080 = 2024-07-18 23:59:20 UTC, 100.0% complete
2024-07-18 11:30:40,282 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 543, 83.8% complete
2024-07-18 11:34:27,042 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-07-18 11:34:27,045 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-18 11:34:27,382 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 11:34:29,727 INFO [download_results] running in process 667082
2024-07-18 11:34:29,728 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:34:29,729 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-18 11:34:29,731 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:34:29,732 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/18jul24 to /results2/SalishSea/nowcast-green.202111
2024-07-18 11:35:40,309 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 630, 97.2% complete
2024-07-18 11:40:40,300 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-18 11:40:40,303 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-18 11:40:40,640 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 11:40:43,097 INFO [download_wwatch3_results] running in process 667173
2024-07-18 11:40:43,098 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:40:43,100 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2024-07-18 11:40:43,101 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:46:33,598 INFO [download_wwatch3_results] forecast 2024-07-18 results files from arbutus.cloud-nowcast downloaded
2024-07-18 11:46:33,602 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-18 11:46:33,780 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 11:46:37,064 INFO [update_forecast_datasets] running in process 667259
2024-07-18 11:46:37,065 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:46:37,067 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2024-07-18 11:46:37,068 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:46:37,070 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/18jul24 run
2024-07-18 11:46:37,072 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/18jul24 run
2024-07-18 11:46:37,121 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/18jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-18 11:46:37,123 INFO [update_forecast_datasets] wwatch3 2024-07-18 forecast rolling forecast datasets updated
2024-07-18 11:46:37,125 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-18 11:46:37,297 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-18 11:46:39,736 INFO [ping_erddap] running in process 667266
2024-07-18 11:46:39,738 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:46:39,740 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-07-18 11:46:39,741 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:46:39,744 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-18 11:46:39,746 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-18 11:46:39,917 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 11:46:47,960 INFO [make_plots] running in process 667272
2024-07-18 11:46:47,962 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 11:46:47,963 INFO [make_plots] publishing log messages to tcp://*:5564
2024-07-18 11:46:47,964 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-18 11:46:47,966 INFO [make_plots] preparing render list for 2024-07-18 WaveWatch3 forecast publish figures
2024-07-18 11:46:47,967 INFO [make_plots] added Halibut Bank figure to 2024-07-18 WaveWatch3 forecast publish render list
2024-07-18 11:46:47,968 INFO [make_plots] added Sentry Shoal figure to 2024-07-18 WaveWatch3 forecast publish render list
2024-07-18 11:46:47,969 INFO [make_plots] starting to render wwatch3 forecast publish 18jul24 figures
2024-07-18 11:46:48,750 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-18 11:46:59,424 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-18 11:47:06,133 INFO [make_plots] finished rendering wwatch3 forecast publish 18jul24 figures
2024-07-18 11:47:06,134 INFO [make_plots] wwatch3 publish plots for 2024-07-18 forecast completed
2024-07-18 11:47:06,138 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-18 12:11:33,096 INFO [download_results] nowcast-green 2024-07-18 results files from arbutus.cloud-nowcast downloaded
2024-07-18 12:11:33,102 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 12:11:33,276 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 12:11:33,280 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-07-18 12:11:33,285 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 12:11:33,289 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 12:11:33,293 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-07-18'], host='localhost')
2024-07-18 12:11:34,597 INFO [ping_erddap] running in process 667696
2024-07-18 12:11:34,599 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 12:11:34,599 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-18 12:11:34,600 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-18 12:11:34,604 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-07-18 12:11:34,607 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-18 12:11:35,585 INFO [make_averaged_dataset] running in process 667697
2024-07-18 12:11:35,586 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 12:11:35,587 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2024-07-18 12:11:35,588 INFO [make_averaged_dataset] running in process 667698
2024-07-18 12:11:35,589 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 12:11:35,589 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2024-07-18 12:11:35,590 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-18 12:11:35,591 INFO [make_averaged_dataset] creating day-averaged dataset for 18-Jul-2024 chemistry
2024-07-18 12:11:35,592 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-18 12:11:35,593 INFO [make_averaged_dataset] creating day-averaged dataset for 18-Jul-2024 biology
2024-07-18 12:11:35,617 INFO [make_averaged_dataset] running in process 667699
2024-07-18 12:11:35,618 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 12:11:35,619 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2024-07-18 12:11:35,620 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-07-18 12:11:35,621 INFO [make_averaged_dataset] creating day-averaged dataset for 18-Jul-2024 physics
2024-07-18 12:11:36,705 INFO [make_plots] running in process 667694
2024-07-18 12:11:36,706 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 12:11:36,707 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-18 12:11:36,708 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-18 12:11:37,287 INFO [make_plots] preparing render list for 2024-07-18 NEMO nowcast-green research figures
2024-07-18 12:11:39,006 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:39,713 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:40,407 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:41,105 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:41,862 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:42,610 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:43,314 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:44,106 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:44,892 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:45,677 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:46,474 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:47,185 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:47,877 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:47,878 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:47,879 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:47,879 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-07-18 NEMO nowcast-green research render list
2024-07-18 12:11:47,880 INFO [make_plots] starting to render nemo nowcast-green research 18jul24 figures
2024-07-18 12:12:07,464 INFO [make_averaged_dataset] day-averaged dataset for 18-Jul-2024 physics created
2024-07-18 12:12:07,467 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-18 12:13:01,141 INFO [make_averaged_dataset] day-averaged dataset for 18-Jul-2024 chemistry created
2024-07-18 12:13:01,144 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-18 12:13:13,974 INFO [make_averaged_dataset] day-averaged dataset for 18-Jul-2024 biology created
2024-07-18 12:13:13,977 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-07-18 12:24:47,925 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 12:24:48,262 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-01'], host='localhost')
2024-07-18 12:24:48,268 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 12:24:48,273 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 12:24:49,784 INFO [download_results] running in process 668004
2024-07-18 12:24:49,785 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 12:24:49,786 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-18 12:24:49,787 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 12:24:49,788 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 12:24:53,918 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 12:25:36,159 INFO [download_results] hindcast 2018-11-01 results files from optimum-hindcast downloaded
2024-07-18 12:25:36,168 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 12:25:36,343 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-01'], host='localhost')
2024-07-18 12:25:37,528 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 13:22:45,973 INFO [make_plots] finished rendering nemo nowcast-green research 18jul24 figures
2024-07-18 13:22:45,983 INFO [make_plots] nemo research plots for 2024-07-18 nowcast-green completed
2024-07-18 13:22:46,040 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-18 13:49:58,582 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 13:49:58,973 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-06'], host='localhost')
2024-07-18 13:49:58,982 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 13:49:58,988 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 13:50:00,823 INFO [download_results] running in process 669104
2024-07-18 13:50:00,824 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 13:50:00,826 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-18 13:50:00,827 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 13:50:00,829 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 13:50:04,922 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 13:50:45,122 INFO [download_results] hindcast 2018-11-06 results files from optimum-hindcast downloaded
2024-07-18 13:50:45,134 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 13:50:45,369 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-06'], host='localhost')
2024-07-18 13:50:46,847 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 15:14:54,012 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240718/18/
2024-07-18 15:14:54,013 INFO [collect_weather] 2024-07-18 2.5km weather forecast 18 collection complete
2024-07-18 15:14:54,017 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-18 15:14:54,389 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-07-18 15:14:54,410 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-07-18 15:14:54,430 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-07-18 15:14:54,453 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-07-19'], host='localhost')
2024-07-18 15:14:54,677 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240718/18/
2024-07-18 15:14:54,679 INFO [crop_gribs] 2024-07-18 18 GRIBs cropping complete
2024-07-18 15:14:54,682 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-18 15:14:56,545 INFO [download_weather] running in process 670356
2024-07-18 15:14:56,552 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 15:14:56,561 INFO [download_weather] publishing log messages to tcp://*:5562
2024-07-18 15:14:56,571 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-18 15:14:56,580 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240718
2024-07-18 15:14:56,581 INFO [download_weather] running in process 670358
2024-07-18 15:14:56,591 INFO [collect_weather] running in process 670359
2024-07-18 15:14:56,592 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 15:14:56,602 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 15:14:56,604 INFO [download_weather] publishing log messages to tcp://*:5565
2024-07-18 15:14:56,605 INFO [collect_weather] publishing log messages to tcp://*:5566
2024-07-18 15:14:56,606 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-18 15:14:56,607 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-18 15:14:56,608 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240718
2024-07-18 15:14:56,612 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-07-18 15:14:58,373 INFO [crop_gribs] running in process 670360
2024-07-18 15:14:58,378 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 15:14:58,379 INFO [crop_gribs] publishing log messages to tcp://*:5567
2024-07-18 15:14:58,380 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-18 15:14:58,381 INFO [crop_gribs] cropping 2024-07-19 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-07-18 15:14:58,407 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240719/00/
2024-07-18 15:15:09,519 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 15:15:09,878 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-11'], host='localhost')
2024-07-18 15:15:09,883 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 15:15:09,887 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 15:15:11,536 INFO [download_results] running in process 670402
2024-07-18 15:15:11,537 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 15:15:11,542 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-18 15:15:11,546 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 15:15:11,548 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 15:15:16,160 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 15:16:01,894 INFO [download_results] hindcast 2018-11-11 results files from optimum-hindcast downloaded
2024-07-18 15:16:01,902 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 15:16:02,131 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-11'], host='localhost')
2024-07-18 15:16:03,924 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 15:20:48,932 INFO [download_weather] 2024-07-18 1km weather forecast 00 downloads complete
2024-07-18 15:20:48,960 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-18 15:20:53,090 INFO [download_weather] 2024-07-18 1km weather forecast 12 downloads complete
2024-07-18 15:20:53,093 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-07-18 16:40:20,724 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 16:40:21,097 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-16'], host='localhost')
2024-07-18 16:40:21,108 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 16:40:21,116 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 16:40:23,433 INFO [download_results] running in process 671834
2024-07-18 16:40:23,434 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 16:40:23,435 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-18 16:40:23,436 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 16:40:23,437 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 16:40:27,735 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 16:41:03,067 INFO [download_results] hindcast 2018-11-16 results files from optimum-hindcast downloaded
2024-07-18 16:41:03,078 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 16:41:03,282 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-16'], host='localhost')
2024-07-18 16:41:04,750 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 18:10:32,342 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 18:10:32,688 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-21'], host='localhost')
2024-07-18 18:10:32,695 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 18:10:32,700 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 18:10:33,940 INFO [download_results] running in process 672842
2024-07-18 18:10:33,941 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 18:10:33,942 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-18 18:10:33,943 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 18:10:33,944 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 18:10:38,125 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 18:11:13,179 INFO [download_results] hindcast 2018-11-21 results files from optimum-hindcast downloaded
2024-07-18 18:11:13,190 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 18:11:13,390 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-21'], host='localhost')
2024-07-18 18:11:14,481 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 19:35:42,556 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 19:35:42,919 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-11-26'], host='localhost')
2024-07-18 19:35:42,930 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 19:35:42,937 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 19:35:44,957 INFO [download_results] running in process 673846
2024-07-18 19:35:44,959 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 19:35:44,961 INFO [download_results] publishing log messages to tcp://*:5564
2024-07-18 19:35:44,963 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 19:35:44,964 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26nov18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 19:35:49,077 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 19:37:25,318 INFO [download_results] hindcast 2018-11-26 results files from optimum-hindcast downloaded
2024-07-18 19:37:25,329 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 19:37:25,534 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-11-26'], host='localhost')
2024-07-18 19:37:27,828 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 19:37:28,036 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-nov', 'robot.graham'], host='localhost')
2024-07-18 19:37:29,528 INFO [archive_tarball] running in process 673912
2024-07-18 19:37:29,529 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 19:37:29,531 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-07-18 19:37:29,532 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-18 19:37:29,534 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-nov18.tar from /ocean/sallen/timbucktwo/oxygen/*nov18/
2024-07-18 19:45:12,281 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-nov18.index from /ocean/dlatorne/oxygen-nov18.tar
2024-07-18 19:45:24,426 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-nov18.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-07-18 19:55:40,625 INFO [archive_tarball] hindcast *nov18 results files archived to robot.graham
2024-07-18 19:55:40,628 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-07-18 21:00:53,775 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 21:00:54,136 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-01'], host='localhost')
2024-07-18 21:00:54,143 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 21:00:54,147 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 21:00:55,411 INFO [download_results] running in process 675002
2024-07-18 21:00:55,412 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 21:00:55,413 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-18 21:00:55,414 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 21:00:55,415 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 21:00:59,721 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 21:01:35,270 INFO [download_results] hindcast 2018-12-01 results files from optimum-hindcast downloaded
2024-07-18 21:01:35,279 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 21:01:35,477 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-01'], host='localhost')
2024-07-18 21:01:36,604 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 21:07:29,288 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240719/00/
2024-07-18 21:07:29,289 INFO [collect_weather] 2024-07-19 2.5km weather forecast 00 collection complete
2024-07-18 21:07:29,292 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-18 21:07:29,660 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-07-18 21:07:29,666 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-07-19'], host='localhost')
2024-07-18 21:07:30,476 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240719/00/
2024-07-18 21:07:30,478 INFO [crop_gribs] 2024-07-19 00 GRIBs cropping complete
2024-07-18 21:07:30,482 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-18 21:07:31,055 INFO [collect_weather] running in process 675172
2024-07-18 21:07:31,057 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 21:07:31,058 INFO [collect_weather] publishing log messages to tcp://*:5562
2024-07-18 21:07:31,058 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-18 21:07:31,077 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-07-18 21:07:32,717 INFO [crop_gribs] running in process 675174
2024-07-18 21:07:32,718 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 21:07:32,719 INFO [crop_gribs] publishing log messages to tcp://*:5564
2024-07-18 21:07:32,720 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-18 21:07:32,721 INFO [crop_gribs] cropping 2024-07-19 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-07-18 21:07:32,735 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240719/06/
2024-07-18 22:26:04,174 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 22:26:04,521 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-06'], host='localhost')
2024-07-18 22:26:04,529 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 22:26:04,534 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 22:26:05,708 INFO [download_results] running in process 676151
2024-07-18 22:26:05,709 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 22:26:05,710 INFO [download_results] publishing log messages to tcp://*:5566
2024-07-18 22:26:05,711 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 22:26:05,713 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 22:26:09,961 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 22:26:45,550 INFO [download_results] hindcast 2018-12-06 results files from optimum-hindcast downloaded
2024-07-18 22:26:45,560 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 22:26:45,760 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-06'], host='localhost')
2024-07-18 22:26:46,802 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-18 23:51:13,985 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-18 23:51:14,338 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-11'], host='localhost')
2024-07-18 23:51:14,356 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 23:51:14,361 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-18 23:51:15,517 INFO [download_results] running in process 677362
2024-07-18 23:51:15,518 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-18 23:51:15,519 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-18 23:51:15,519 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-18 23:51:15,520 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-18 23:51:19,798 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-18 23:51:55,397 INFO [download_results] hindcast 2018-12-11 results files from optimum-hindcast downloaded
2024-07-18 23:51:55,406 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-18 23:51:55,611 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-11'], host='localhost')
2024-07-18 23:51:56,717 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-19 01:16:24,056 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-19 01:16:24,438 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-16'], host='localhost')
2024-07-19 01:16:24,445 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-19 01:16:24,450 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-19 01:16:25,693 INFO [download_results] running in process 678498
2024-07-19 01:16:25,694 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 01:16:25,696 INFO [download_results] publishing log messages to tcp://*:5565
2024-07-19 01:16:25,697 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-19 01:16:25,698 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-19 01:16:29,893 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-19 01:17:05,679 INFO [download_results] hindcast 2018-12-16 results files from optimum-hindcast downloaded
2024-07-19 01:17:05,689 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-19 01:17:05,898 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-16'], host='localhost')
2024-07-19 01:17:07,034 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-19 02:41:34,295 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-19 02:41:34,653 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-21'], host='localhost')
2024-07-19 02:41:34,660 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-19 02:41:34,664 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-19 02:41:35,859 INFO [download_results] running in process 678903
2024-07-19 02:41:35,861 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 02:41:35,862 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-19 02:41:35,863 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-19 02:41:35,864 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-19 02:41:40,057 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-19 02:42:15,793 INFO [download_results] hindcast 2018-12-21 results files from optimum-hindcast downloaded
2024-07-19 02:42:15,803 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-19 02:42:16,008 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-21'], host='localhost')
2024-07-19 02:42:17,079 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-19 03:02:52,805 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240719/06/
2024-07-19 03:02:52,806 INFO [collect_weather] 2024-07-19 2.5km weather forecast 06 collection complete
2024-07-19 03:02:52,809 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-07-19 03:02:53,172 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,178 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,182 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,186 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,192 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,196 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,200 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,206 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,210 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,213 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,225 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,231 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,238 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,245 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,267 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-07-19 03:02:53,281 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-07-19 03:02:53,298 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-07-19 03:02:53,311 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2024-07-18'], host='localhost')
2024-07-19 03:02:53,329 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-07-19 03:02:53,335 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-07-19 03:02:53,368 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-07-19 03:02:54,550 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240719/06/
2024-07-19 03:02:54,551 INFO [crop_gribs] 2024-07-19 06 GRIBs cropping complete
2024-07-19 03:02:54,552 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-07-19 03:02:55,269 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-07-19 03:02:56,323 INFO [collect_NeahBay_ssh] running in process 679014
2024-07-19 03:02:56,323 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:56,324 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2024-07-19 03:02:56,325 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:56,326 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-07-19
2024-07-19 03:02:56,339 INFO [collect_weather] running in process 679015
2024-07-19 03:02:56,341 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:56,342 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-07-19 03:02:56,343 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:56,365 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-07-19 03:02:57,210 INFO [collect_river_data] running in process 678994
2024-07-19 03:02:57,211 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,212 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-07-19 03:02:57,213 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,214 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-07-18
2024-07-19 03:02:57,245 INFO [collect_river_data] running in process 679004
2024-07-19 03:02:57,246 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,247 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-07-19 03:02:57,248 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,248 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-07-18
2024-07-19 03:02:57,286 INFO [collect_river_data] running in process 679002
2024-07-19 03:02:57,288 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,289 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-07-19 03:02:57,290 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,291 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-07-18
2024-07-19 03:02:57,313 INFO [collect_river_data] running in process 679003
2024-07-19 03:02:57,314 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,315 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-07-19 03:02:57,316 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,317 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-07-18
2024-07-19 03:02:57,346 INFO [collect_river_data] running in process 679006
2024-07-19 03:02:57,349 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,350 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-07-19 03:02:57,352 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,353 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-07-18
2024-07-19 03:02:57,390 INFO [collect_river_data] running in process 678999
2024-07-19 03:02:57,391 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,392 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-07-19 03:02:57,393 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,394 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-07-18
2024-07-19 03:02:57,415 INFO [collect_river_data] running in process 678998
2024-07-19 03:02:57,417 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,418 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-07-19 03:02:57,419 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,420 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-07-18
2024-07-19 03:02:57,448 INFO [collect_river_data] running in process 679000
2024-07-19 03:02:57,449 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,450 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-07-19 03:02:57,452 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,453 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-07-18
2024-07-19 03:02:57,454 INFO [collect_river_data] running in process 679005
2024-07-19 03:02:57,455 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,455 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-07-19 03:02:57,456 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,457 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-07-18
2024-07-19 03:02:57,492 INFO [collect_river_data] running in process 679008
2024-07-19 03:02:57,494 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,495 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-07-19 03:02:57,496 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,497 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-07-18
2024-07-19 03:02:57,508 INFO [collect_river_data] running in process 678996
2024-07-19 03:02:57,509 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,510 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-07-19 03:02:57,511 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,512 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-07-18
2024-07-19 03:02:57,523 INFO [collect_river_data] running in process 679007
2024-07-19 03:02:57,524 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,525 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-07-19 03:02:57,526 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,526 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-07-18
2024-07-19 03:02:57,562 INFO [collect_river_data] running in process 678997
2024-07-19 03:02:57,565 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,566 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-07-19 03:02:57,567 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,567 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-07-18
2024-07-19 03:02:57,640 INFO [collect_river_data] running in process 679009
2024-07-19 03:02:57,641 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,642 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-07-19 03:02:57,643 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,644 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-07-18
2024-07-19 03:02:57,763 INFO [get_vfpa_hadcp] running in process 679013
2024-07-19 03:02:57,764 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:57,765 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5590
2024-07-19 03:02:57,766 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:57,767 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-18
2024-07-19 03:02:57,812 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-07-18 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-07-19 03:02:57,813 INFO [collect_river_data] ECCC Capilano river data collection for 2024-07-18 completed
2024-07-19 03:02:57,815 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:02:57,865 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-07-19 03:02:57,866 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-07-18 completed
2024-07-19 03:02:57,888 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-07-19 03:02:57,888 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-07-18 completed
2024-07-19 03:02:57,903 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-07-19 03:02:57,903 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-07-18 completed
2024-07-19 03:02:57,953 INFO [collect_NeahBay_ssh] 2024-07-19T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-07-19 03:02:57,969 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-07-18 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-07-19 03:02:57,970 INFO [collect_river_data] ECCC Englishman river data collection for 2024-07-18 completed
2024-07-19 03:02:58,002 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-07-18 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-07-19 03:02:58,002 INFO [collect_river_data] ECCC Fraser river data collection for 2024-07-18 completed
2024-07-19 03:02:58,039 INFO [crop_gribs] running in process 679016
2024-07-19 03:02:58,041 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:58,042 INFO [crop_gribs] publishing log messages to tcp://*:5591
2024-07-19 03:02:58,043 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:58,044 INFO [crop_gribs] cropping 2024-07-19 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-07-19 03:02:58,062 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-07-19 03:02:58,063 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-07-18 completed
2024-07-19 03:02:58,066 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240719/12/
2024-07-19 03:02:58,116 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-07-19 03:02:58,117 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-07-18 completed
2024-07-19 03:02:58,237 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-07-19 03:02:58,238 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-07-18 completed
2024-07-19 03:02:58,270 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-07-19 03:02:58,271 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-07-18 completed
2024-07-19 03:02:58,273 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:02:58,292 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-07-19 03:02:58,294 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-07-18 completed
2024-07-19 03:02:58,299 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-07-19 03:02:58,300 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-07-18 completed
2024-07-19 03:02:58,342 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-07-19 03:02:58,344 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-07-18 completed
2024-07-19 03:02:58,355 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-07-18 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-07-19 03:02:58,358 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-07-18 completed
2024-07-19 03:02:58,528 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:02:58,744 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:02:58,795 INFO [get_onc_ferry] running in process 679012
2024-07-19 03:02:58,796 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:58,797 INFO [get_onc_ferry] publishing log messages to tcp://*:5566
2024-07-19 03:02:58,797 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:58,798 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-07-18
2024-07-19 03:02:58,804 INFO [get_onc_ctd] running in process 679011
2024-07-19 03:02:58,805 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:58,806 INFO [get_onc_ctd] publishing log messages to tcp://*:5568
2024-07-19 03:02:58,808 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:58,809 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-07-18
2024-07-19 03:02:58,838 INFO [get_onc_ctd] running in process 679010
2024-07-19 03:02:58,839 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:58,840 INFO [get_onc_ctd] publishing log messages to tcp://*:5592
2024-07-19 03:02:58,842 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:58,842 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-07-18
2024-07-19 03:02:58,862 INFO [grib_to_netcdf] running in process 679277
2024-07-19 03:02:58,863 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:02:58,864 INFO [grib_to_netcdf] publishing log messages to tcp://*:5593
2024-07-19 03:02:58,864 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:02:58,865 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-07-19 forecast2 run
2024-07-19 03:02:58,937 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-07-19 03:02:59,122 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-07-19'], host='localhost')
2024-07-19 03:02:59,128 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:02:59,363 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:02:59,586 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:02:59,819 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:03:00,034 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:03:00,228 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:03:00,424 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:03:00,476 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2024-07-18 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202407.nc
2024-07-19 03:03:00,477 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2024-07 netcdf file
2024-07-19 03:03:00,638 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:03:00,824 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:03:00,902 ERROR [get_onc_ctd] No ONC SCVIP CTD T&S data for 2024-07-18
2024-07-19 03:03:01,016 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-07-19 03:03:01,207 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2024-07-19 03:03:01,405 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2024-07-19 03:03:01,499 CRITICAL [get_onc_ctd] 2024-07-18 ONC SCVIP CTD T&S file creation failed
2024-07-19 03:03:02,197 INFO [make_ssh_files] running in process 679456
2024-07-19 03:03:02,198 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:02,199 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2024-07-19 03:03:02,201 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:02,202 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-07-19 NOAA Neah Bay 00Z observation and forecast values
2024-07-19 03:03:02,592 INFO [ping_erddap] running in process 679489
2024-07-19 03:03:02,593 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:02,596 INFO [ping_erddap] publishing log messages to tcp://*:5567
2024-07-19 03:03:02,598 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:02,601 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2024-07-19 03:03:02,605 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-19 03:03:06,988 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d14.nc
2024-07-19 03:03:07,174 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d15.nc
2024-07-19 03:03:07,349 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d16.nc
2024-07-19 03:03:07,530 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d17.nc
2024-07-19 03:03:07,696 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d20.nc
2024-07-19 03:03:07,835 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m07d18.nc
2024-07-19 03:03:08,131 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d19.nc
2024-07-19 03:03:08,346 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d20.nc
2024-07-19 03:03:08,523 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d21.nc
2024-07-19 03:03:08,699 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m07d22.nc
2024-07-19 03:03:08,863 INFO [make_ssh_files] sea surface height boundary file for 2024-07-19 forecast2 run created
2024-07-19 03:03:08,867 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-07-19 03:03:09,076 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-07-19 03:03:09,080 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-07-19 03:03:10,030 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m07d21.nc
2024-07-19 03:03:10,036 INFO [grib_to_netcdf] 2024-07-19 NEMO-atmos forcing file for forecast2 created
2024-07-19 03:03:10,038 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-07-19 03:03:10,465 INFO [make_runoff_file] running in process 679529
2024-07-19 03:03:10,466 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:10,467 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2024-07-19 03:03:10,468 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:10,511 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-07-19 03:03:10,618 INFO [make_runoff_file] 2024-07-19 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-07-19 03:03:10,620 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-07-19 03:03:11,122 INFO [make_v202111_runoff_file] running in process 679527
2024-07-19 03:03:11,123 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:11,124 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5562
2024-07-19 03:03:11,125 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:11,126 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-07-18
2024-07-19 03:03:12,105 ERROR [make_v202111_runoff_file] no 2024-07-18 discharge obs for Skagit_MountVernon: patching
2024-07-19 03:03:12,590 ERROR [make_v202111_runoff_file] no 2024-07-18 discharge obs for Snohomish_Monroe: patching
2024-07-19 03:03:12,795 ERROR [make_v202111_runoff_file] no 2024-07-18 discharge obs for Nisqually_McKenna: patching
2024-07-19 03:03:12,891 ERROR [make_v202111_runoff_file] no 2024-07-18 discharge obs for Greenwater_Greenwater: patching
2024-07-19 03:03:14,327 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-07-18: /results/forcing/rivers/R202108Dailies_y2024m07d18.nc
2024-07-19 03:03:14,328 INFO [make_v202111_runoff_file] 2024-07-18 runoff file creation completed
2024-07-19 03:03:14,330 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-07-19 03:03:14,511 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-07-19 03:03:14,516 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-07-19 03:03:14,519 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-07-19 03:03:14,524 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-07-19 03:03:15,825 INFO [upload_forcing] running in process 679570
2024-07-19 03:03:15,826 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:15,827 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-07-19 03:03:15,828 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:15,864 INFO [upload_forcing] running in process 679569
2024-07-19 03:03:15,866 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:15,867 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-07-19 03:03:15,868 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:15,869 INFO [upload_forcing] running in process 679571
2024-07-19 03:03:15,870 INFO [upload_forcing] running in process 679568
2024-07-19 03:03:15,871 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:15,872 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:15,872 INFO [upload_forcing] publishing log messages to tcp://*:5569
2024-07-19 03:03:15,873 INFO [upload_forcing] publishing log messages to tcp://*:5570
2024-07-19 03:03:15,874 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:15,875 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:17,221 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m07d18.nc
2024-07-19 03:03:17,311 INFO [upload_forcing] forecast2 2024-07-19 forcing files upload to optimum-hindcast completed
2024-07-19 03:03:17,314 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-19 03:03:17,450 INFO [upload_forcing] forecast2 2024-07-19 forcing files upload to orcinus-nowcast-agrif completed
2024-07-19 03:03:17,525 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-19 03:03:18,329 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-07-18
2024-07-19 03:03:18,955 INFO [upload_forcing] forecast2 2024-07-19 forcing files upload to arbutus.cloud-nowcast completed
2024-07-19 03:03:18,959 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-19 03:03:19,165 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-19'], host='localhost')
2024-07-19 03:03:19,187 WARNING [get_onc_ferry] No ONC TWDP TSG temperature,conductivity,salinity data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:19,193 WARNING [get_onc_ferry] No ONC TWDP TSG temperature data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:19,194 WARNING [get_onc_ferry] No ONC TWDP TSG conductivity data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:19,196 WARNING [get_onc_ferry] No ONC TWDP TSG salinity data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:19,197 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-07-18
2024-07-19 03:03:19,274 WARNING [get_onc_ctd] no SEVIP salinity data
2024-07-19 03:03:19,523 INFO [get_onc_ctd] 2024-07-18 ONC SEVIP CTD T&S file created
2024-07-19 03:03:19,525 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-07-19 03:03:19,697 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-07-19 03:03:20,050 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation,oxygen_corrected,temperature data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:20,056 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:20,057 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:20,058 WARNING [get_onc_ferry] No ONC TWDP OXYSENSOR temperature data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:20,059 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-07-18
2024-07-19 03:03:20,324 INFO [make_forcing_links] running in process 679598
2024-07-19 03:03:20,325 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:20,327 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2024-07-19 03:03:20,328 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:20,768 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence,chlorophyll,turbidity data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:20,774 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:20,775 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL chlorophyll data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:20,776 WARNING [get_onc_ferry] No ONC TWDP TURBCHLFL turbidity data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:20,777 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-07-18
2024-07-19 03:03:20,828 INFO [ping_erddap] running in process 679600
2024-07-19 03:03:20,830 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:20,831 INFO [ping_erddap] publishing log messages to tcp://*:5568
2024-07-19 03:03:20,832 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:20,834 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-07-19 03:03:20,836 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-19 03:03:21,483 INFO [make_forcing_links] forecast2 2024-07-19 forcing file links on arbutus.cloud-nowcast created
2024-07-19 03:03:21,487 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-07-19 03:03:21,667 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast')
2024-07-19 03:03:21,755 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:21,760 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:21,761 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:21,762 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-07-18
2024-07-19 03:03:23,194 INFO [upload_forcing] forecast2 2024-07-19 forcing files upload to robot.graham completed
2024-07-19 03:03:23,199 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-07-19 03:03:23,767 INFO [run_NEMO] running in process 866
2024-07-19 03:03:23,769 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:23,770 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-07-19 03:03:26,367 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:27,305 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/18jul24forecast2_2024-07-19T030326.478187-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-07-19 03:03:27,324 INFO [run_NEMO] forecast2 NEMO run for 2024-07-19 on arbutus.cloud-nowcast started
2024-07-19 03:03:27,329 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-07-19 03:03:27,537 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-19 03:03:29,492 INFO [watch_NEMO] running in process 1126
2024-07-19 03:03:29,494 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:03:29,495 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-07-19 03:03:29,496 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:03:29,566 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-07-19 03:03:39,477 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-07-18
2024-07-19 03:03:47,934 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-07-18
2024-07-19 03:03:56,119 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-07-18; substituting empty dataset
2024-07-19 03:03:56,119 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-07-18
2024-07-19 03:04:03,909 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-18; substituting empty dataset
2024-07-19 03:04:03,911 WARNING [get_onc_ferry] ONC TWDP TSG temperature data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:03,932 WARNING [get_onc_ferry] ONC TWDP TSG conductivity data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:03,995 WARNING [get_onc_ferry] ONC TWDP TSG salinity data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,013 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_saturation data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,030 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR oxygen_corrected data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,048 WARNING [get_onc_ferry] ONC TWDP OXYSENSOR temperature data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,065 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL cdom_fluorescence data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,083 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL chlorophyll data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,100 WARNING [get_onc_ferry] ONC TWDP TURBCHLFL turbidity data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,118 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,135 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,239 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:04:04,256 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-07-18 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-07-19 03:05:10,403 INFO [get_onc_ferry] 2024-07-18 ONC TWDP ferry data file created
2024-07-19 03:05:10,407 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-07-19 03:05:10,608 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-07-19 03:05:11,853 INFO [ping_erddap] running in process 679623
2024-07-19 03:05:11,854 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:05:11,856 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-19 03:05:11,857 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:05:11,859 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-07-19 03:05:11,861 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-19 03:08:29,553 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17796697 = 2024-07-20 05:04:00 UTC, 16.9% complete
2024-07-19 03:13:29,544 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17797183 = 2024-07-20 10:28:00 UTC, 34.9% complete
2024-07-19 03:18:29,538 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17797674 = 2024-07-20 15:55:20 UTC, 53.1% complete
2024-07-19 03:23:29,533 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17798144 = 2024-07-20 21:08:40 UTC, 70.5% complete
2024-07-19 03:28:29,530 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17798594 = 2024-07-21 02:08:40 UTC, 87.2% complete
2024-07-19 03:33:29,526 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17798940 = 2024-07-21 05:59:20 UTC, 100.0% complete
2024-07-19 03:38:29,526 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-07-19 03:38:29,530 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-07-19 03:38:29,886 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast')
2024-07-19 03:38:29,894 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast')
2024-07-19 03:38:29,898 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-18'], host='localhost')
2024-07-19 03:38:31,051 INFO [download_results] running in process 680096
2024-07-19 03:38:31,052 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:38:31,053 INFO [download_results] publishing log messages to tcp://*:5562
2024-07-19 03:38:31,054 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:38:31,055 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/18jul24 to /results/SalishSea/forecast2.202111
2024-07-19 03:38:32,847 INFO [make_ww3_wind_file] running in process 2272
2024-07-19 03:38:32,848 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:38:32,849 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2024-07-19 03:38:32,850 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:38:32,851 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2024-07-19 forecast2 run
2024-07-19 03:38:32,866 INFO [make_ww3_current_file] running in process 2271
2024-07-19 03:38:32,868 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 03:38:32,869 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2024-07-19 03:38:32,870 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2024-07-19 03:38:32,871 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2024-07-19 forecast2 run
2024-07-19 03:38:42,196 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2024-07-19 forecast2 run
2024-07-19 03:38:42,200 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-07-19 03:40:42,373 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2024-07-19 forecast2 run
2024-07-19 03:40:42,378 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-07-19 03:40:42,587 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-19'], host='arbutus.cloud-nowcast')
2024-07-19 03:40:52,430 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-07-19 03:40:52,639 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-07-19 03:40:59,943 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-07-19 03:45:59,742 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 84, 15.6% complete
2024-07-19 03:51:00,243 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 174, 32.2% complete
2024-07-19 03:55:59,784 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 252, 46.7% complete
2024-07-19 04:01:00,069 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 327, 60.6% complete
2024-07-19 04:03:32,133 INFO [download_results] forecast2 2024-07-18 results files from arbutus.cloud-nowcast downloaded
2024-07-19 04:03:32,137 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-19 04:03:32,340 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-07-18'], host='localhost')
2024-07-19 04:03:34,021 INFO [make_CHS_currents_file] running in process 680167
2024-07-19 04:03:34,022 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:03:34,023 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-07-19 04:03:34,024 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:04:08,197 INFO [make_CHS_currents_file] Made CHS currents file for 2024-07-18 for forecast2
2024-07-19 04:04:08,200 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-07-19 04:04:08,360 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-07-18'], host='localhost')
2024-07-19 04:04:09,364 INFO [update_forecast_datasets] running in process 680225
2024-07-19 04:04:09,365 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:04:09,366 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-07-19 04:04:09,367 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:04:09,368 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/18jul24 run
2024-07-19 04:05:59,569 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 417, 77.2% complete
2024-07-19 04:08:20,297 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/18jul24 run: /results/SalishSea/rolling-forecasts/nemo
2024-07-19 04:08:20,298 INFO [update_forecast_datasets] nemo 2024-07-18 forecast2 rolling forecast datasets updated
2024-07-19 04:08:20,301 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-19 04:08:20,467 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-07-19 04:08:20,473 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-07-18'], host='localhost')
2024-07-19 04:08:20,478 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-07-18'], host='localhost')
2024-07-19 04:08:21,759 INFO [ping_erddap] running in process 680303
2024-07-19 04:08:21,760 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:08:21,761 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-07-19 04:08:21,762 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:08:21,772 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-07-19 04:08:21,774 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-19 04:08:22,980 INFO [make_surface_current_tiles] running in process 680306
2024-07-19 04:08:22,982 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:08:22,983 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2024-07-19 04:08:22,984 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:08:22,986 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-07-18 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/18jul24
2024-07-19 04:08:26,408 INFO [make_plots] running in process 680305
2024-07-19 04:08:26,413 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:08:26,414 INFO [make_plots] publishing log messages to tcp://*:5566
2024-07-19 04:08:26,415 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:08:27,147 INFO [make_plots] preparing render list for 2024-07-18 NEMO forecast2 publish figures
2024-07-19 04:08:27,270 INFO [make_plots] added Website_thumbnail figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,271 INFO [make_plots] added Threshold_website figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,272 INFO [make_plots] added PA_tidal_predictions figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,273 INFO [make_plots] added Boundary Bay figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,273 INFO [make_plots] added Campbell River figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,274 INFO [make_plots] added Cherry Point figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,274 INFO [make_plots] added Friday Harbor figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,275 INFO [make_plots] added Halfmoon Bay figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,276 INFO [make_plots] added Nanaimo figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,277 INFO [make_plots] added Neah Bay figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,278 INFO [make_plots] added New Westminster figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,279 INFO [make_plots] added Patricia Bay figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,279 INFO [make_plots] added Point Atkinson figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,280 INFO [make_plots] added Port Renfrew figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,281 INFO [make_plots] added Sand Heads figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,282 INFO [make_plots] added Sandy Cove figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,283 INFO [make_plots] added Squamish figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,285 INFO [make_plots] added Victoria figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,286 INFO [make_plots] added Woodwards Landing figure to 2024-07-18 NEMO forecast2 publish render list
2024-07-19 04:08:27,287 INFO [make_plots] starting to render nemo forecast2 publish 18jul24 figures
2024-07-19 04:10:59,566 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 504, 93.3% complete
2024-07-19 04:15:59,563 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-07-19 04:15:59,566 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-07-19 04:16:00,051 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-07-19'], host='localhost')
2024-07-19 04:16:01,489 INFO [download_wwatch3_results] running in process 681730
2024-07-19 04:16:01,491 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:16:01,491 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2024-07-19 04:16:01,492 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:18:15,422 INFO [download_wwatch3_results] forecast2 2024-07-19 results files from arbutus.cloud-nowcast downloaded
2024-07-19 04:18:15,425 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-07-19 04:18:15,820 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-07-19'], host='localhost')
2024-07-19 04:18:17,413 INFO [update_forecast_datasets] running in process 682136
2024-07-19 04:18:17,414 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:18:17,415 INFO [update_forecast_datasets] publishing log messages to tcp://*:5567
2024-07-19 04:18:17,416 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:18:17,416 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/19jul24 run
2024-07-19 04:18:17,417 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/19jul24 run
2024-07-19 04:18:52,293 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/19jul24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-07-19 04:18:52,294 INFO [update_forecast_datasets] wwatch3 2024-07-19 forecast2 rolling forecast datasets updated
2024-07-19 04:18:52,296 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-07-19 04:18:52,488 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-07-19 04:18:54,003 INFO [ping_erddap] running in process 682240
2024-07-19 04:18:54,004 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:18:54,005 INFO [ping_erddap] publishing log messages to tcp://*:5567
2024-07-19 04:18:54,006 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:18:54,007 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-07-19 04:18:54,009 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-07-19 04:18:54,210 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-07-19'], host='localhost')
2024-07-19 04:18:58,965 INFO [make_plots] running in process 682255
2024-07-19 04:18:58,966 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:18:58,967 INFO [make_plots] publishing log messages to tcp://*:5567
2024-07-19 04:18:58,969 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:18:58,970 INFO [make_plots] preparing render list for 2024-07-19 WaveWatch3 forecast2 publish figures
2024-07-19 04:18:58,971 INFO [make_plots] added Halibut Bank figure to 2024-07-19 WaveWatch3 forecast2 publish render list
2024-07-19 04:18:58,972 INFO [make_plots] added Sentry Shoal figure to 2024-07-19 WaveWatch3 forecast2 publish render list
2024-07-19 04:18:58,973 INFO [make_plots] starting to render wwatch3 forecast2 publish 19jul24 figures
2024-07-19 04:18:59,464 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-07-19 04:19:08,683 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-07-19 04:19:15,326 INFO [make_plots] finished rendering wwatch3 forecast2 publish 19jul24 figures
2024-07-19 04:19:15,327 INFO [make_plots] wwatch3 publish plots for 2024-07-19 forecast2 completed
2024-07-19 04:19:15,330 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-19 04:19:42,517 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/18jul24
2024-07-19 04:21:37,667 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-07-19 04:21:37,668 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-07-19 04:22:13,482 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:23:53,690 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/18jul24
2024-07-19 04:23:53,702 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-07-18 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/18jul24
2024-07-19 04:23:53,703 INFO [make_surface_current_tiles] surface current tile figures for 2024-07-18 forecast2 completed
2024-07-19 04:23:53,969 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-07-19 04:24:02,002 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:24:02,351 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:24:37,699 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:25:48,453 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:26:24,751 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:26:46,457 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-07-19 04:26:47,005 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2018-12-26'], host='localhost')
2024-07-19 04:26:47,011 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-19 04:26:47,015 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-07-19 04:26:48,221 INFO [download_results] running in process 682392
2024-07-19 04:26:48,222 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:26:48,223 INFO [download_results] publishing log messages to tcp://*:5567
2024-07-19 04:26:48,224 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:26:48,226 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26dec18 to /ocean/sallen/timbucktwo/oxygen
2024-07-19 04:26:52,451 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-07-19 04:27:02,367 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:27:33,042 INFO [download_results] hindcast 2018-12-26 results files from optimum-hindcast downloaded
2024-07-19 04:27:33,051 INFO [manager] checklist updated with [results files] items from download_results worker
2024-07-19 04:27:33,395 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2018-12-26'], host='localhost')
2024-07-19 04:27:34,551 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-07-19 04:27:34,863 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2018-dec', 'robot.graham'], host='localhost')
2024-07-19 04:27:35,932 INFO [archive_tarball] running in process 682439
2024-07-19 04:27:35,934 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:27:35,935 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-07-19 04:27:35,936 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:27:35,937 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-dec18.tar from /ocean/sallen/timbucktwo/oxygen/*dec18/
2024-07-19 04:27:42,383 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:28:20,558 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:28:59,739 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:29:38,595 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:29:38,947 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:30:17,775 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:30:55,316 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-07-14 00:00:00Z to 2024-07-21 05:55:00Z
2024-07-19 04:31:34,302 INFO [make_plots] finished rendering nemo forecast2 publish 18jul24 figures
2024-07-19 04:31:34,304 INFO [make_plots] nemo publish plots for 2024-07-18 forecast2 completed
2024-07-19 04:31:34,314 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-07-19 04:31:34,625 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-07-18'], host='localhost')
2024-07-19 04:31:39,488 INFO [make_feeds] running in process 682490
2024-07-19 04:31:39,490 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:31:39,491 INFO [make_feeds] publishing log messages to tcp://*:5566
2024-07-19 04:31:39,492 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:32:09,979 INFO [make_feeds] ATOM feeds for 2024-07-18 forecast2 run completed
2024-07-19 04:32:09,983 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-07-19 04:32:10,315 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-07-19 04:32:11,346 INFO [clear_checklist] running in process 682517
2024-07-19 04:32:11,348 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:32:11,349 INFO [clear_checklist] publishing log messages to tcp://*:5566
2024-07-19 04:32:11,350 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:32:11,351 INFO [clear_checklist] requesting that manager clear system state checklist
2024-07-19 04:32:11,353 INFO [manager] writing checklist to log file
2024-07-19 04:32:11,406 INFO [manager] checklist cleared
2024-07-19 04:32:11,408 INFO [clear_checklist] nowcast system checklist cleared
2024-07-19 04:32:11,411 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-07-19 04:32:12,474 INFO [rotate_logs] running in process 682522
2024-07-19 04:32:12,475 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-07-19 04:32:12,476 INFO [rotate_logs] publishing log messages to tcp://*:5566
2024-07-19 04:32:12,477 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-07-19 04:32:12,478 INFO [rotate_logs] rotating log files