2024-04-21 04:46:05,826 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-04-21 04:46:05,836 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-04-21 04:46:05,841 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-04-21 04:46:05,843 INFO [rotate_logs] log files rotated
2024-04-21 04:46:05,848 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-04-21 04:55:33,562 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 04:55:33,819 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-12-06'], host='localhost')
2024-04-21 04:55:33,826 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 04:55:33,830 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 04:55:34,928 INFO [download_results] running in process 2082874
2024-04-21 04:55:34,929 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 04:55:34,930 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-21 04:55:34,931 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 04:55:34,932 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06dec19 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 04:55:39,083 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 04:56:28,237 INFO [download_results] hindcast 2019-12-06 results files from optimum-hindcast downloaded
2024-04-21 04:56:28,253 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 04:56:28,262 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-12-06'], host='localhost')
2024-04-21 04:56:29,432 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 06:20:43,624 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 06:20:43,916 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-12-11'], host='localhost')
2024-04-21 06:20:44,084 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 06:20:44,126 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 06:20:45,582 INFO [download_results] running in process 2083850
2024-04-21 06:20:45,583 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 06:20:45,584 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-21 06:20:45,585 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 06:20:45,586 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11dec19 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 06:20:49,797 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 06:21:49,136 INFO [download_results] hindcast 2019-12-11 results files from optimum-hindcast downloaded
2024-04-21 06:21:49,152 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 06:21:49,160 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-12-11'], host='localhost')
2024-04-21 06:21:50,389 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 07:45:54,155 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 07:45:54,363 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-12-16'], host='localhost')
2024-04-21 07:45:54,370 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 07:45:54,376 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 07:45:55,757 INFO [download_results] running in process 2084273
2024-04-21 07:45:55,758 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 07:45:55,759 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-21 07:45:55,761 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 07:45:55,762 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16dec19 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 07:45:59,933 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 07:48:11,258 INFO [download_results] hindcast 2019-12-16 results files from optimum-hindcast downloaded
2024-04-21 07:48:11,270 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 07:48:11,277 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-12-16'], host='localhost')
2024-04-21 07:48:12,523 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 08:03:44,533 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-21 08:03:44,547 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 08:03:46,468 INFO [run_ww3] running in process 26047
2024-04-21 08:03:46,489 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 08:03:46,491 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-21 08:03:46,492 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-21 08:03:46,497 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2024-04-21T080346.466276-0700
2024-04-21 08:03:46,499 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2024-04-21T080346.466276-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-21 08:03:46,501 INFO [run_ww3] forecast2 WaveWatch3 run for 2024-04-21 on arbutus.cloud-nowcast started
2024-04-21 08:03:47,829 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-21 08:03:47,866 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-21 08:03:49,776 INFO [watch_ww3] running in process 26137
2024-04-21 08:03:49,776 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 08:03:49,778 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-04-21 08:03:49,779 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-21 08:03:50,015 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-21 08:08:50,016 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2024-04-21 08:13:50,017 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 189, 35.0% complete
2024-04-21 08:18:50,017 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 288, 53.3% complete
2024-04-21 08:23:50,018 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 381, 70.6% complete
2024-04-21 08:28:50,019 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 477, 88.3% complete
2024-04-21 08:33:50,019 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-21 08:38:50,021 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-21 08:38:50,025 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-21 08:38:50,241 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 08:38:51,265 INFO [download_wwatch3_results] running in process 2084883
2024-04-21 08:38:51,266 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 08:38:51,268 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-04-21 08:38:51,269 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 08:41:08,666 INFO [download_wwatch3_results] forecast2 2024-04-21 results files from arbutus.cloud-nowcast downloaded
2024-04-21 08:41:08,670 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-21 08:41:08,683 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 08:41:09,589 INFO [update_forecast_datasets] running in process 2084894
2024-04-21 08:41:09,590 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 08:41:09,592 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-04-21 08:41:09,592 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-21 08:41:09,593 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/21apr24 run
2024-04-21 08:41:09,594 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/21apr24 run
2024-04-21 08:41:32,779 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/21apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-21 08:41:32,781 INFO [update_forecast_datasets] wwatch3 2024-04-21 forecast2 rolling forecast datasets updated
2024-04-21 08:41:32,783 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-21 08:41:32,796 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-21 08:41:33,689 INFO [ping_erddap] running in process 2084905
2024-04-21 08:41:33,690 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 08:41:33,691 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-04-21 08:41:33,692 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-21 08:41:33,694 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-21 08:41:33,696 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-21 08:41:33,708 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 08:41:39,638 INFO [make_plots] running in process 2084910
2024-04-21 08:41:39,639 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 08:41:39,640 INFO [make_plots] publishing log messages to tcp://*:5565
2024-04-21 08:41:39,641 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-21 08:41:39,642 INFO [make_plots] preparing render list for 2024-04-21 WaveWatch3 forecast2 publish figures
2024-04-21 08:41:39,643 INFO [make_plots] added Halibut Bank figure to 2024-04-21 WaveWatch3 forecast2 publish render list
2024-04-21 08:41:39,644 INFO [make_plots] added Sentry Shoal figure to 2024-04-21 WaveWatch3 forecast2 publish render list
2024-04-21 08:41:39,646 INFO [make_plots] starting to render wwatch3 forecast2 publish 21apr24 figures
2024-04-21 08:41:39,916 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-21 08:41:46,957 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-21 08:41:52,502 INFO [make_plots] finished rendering wwatch3 forecast2 publish 21apr24 figures
2024-04-21 08:41:52,503 INFO [make_plots] wwatch3 publish plots for 2024-04-21 forecast2 completed
2024-04-21 08:41:52,506 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-21 09:03:04,223 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240421/12/
2024-04-21 09:03:04,224 INFO [collect_weather] 2024-04-21 2.5km weather forecast 12 collection complete
2024-04-21 09:03:04,227 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-21 09:03:04,435 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 09:03:04,441 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 09:03:04,445 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 09:03:04,449 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-04-20'], host='localhost')
2024-04-21 09:03:04,453 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-04-21 09:03:04,457 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-04-21 09:03:04,461 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-04-21 09:03:04,465 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-04-21 09:03:04,474 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-04-21 09:03:05,655 INFO [collect_NeahBay_ssh] running in process 2085200
2024-04-21 09:03:05,656 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:05,656 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2024-04-21 09:03:05,657 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:05,658 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-04-21
2024-04-21 09:03:05,753 INFO [collect_weather] running in process 2085202
2024-04-21 09:03:05,754 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:05,755 INFO [collect_weather] publishing log messages to tcp://*:5565
2024-04-21 09:03:05,755 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:05,772 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-04-21 09:03:05,865 INFO [download_live_ocean] running in process 2085201
2024-04-21 09:03:05,865 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:05,866 INFO [download_live_ocean] publishing log messages to tcp://*:5566
2024-04-21 09:03:05,867 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:05,868 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-04-21
2024-04-21 09:03:06,253 INFO [make_turbidity_file] running in process 2085199
2024-04-21 09:03:06,255 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:06,256 INFO [make_turbidity_file] publishing log messages to tcp://*:5567
2024-04-21 09:03:06,257 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:06,258 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-04-21
2024-04-21 09:03:06,546 INFO [collect_river_data] running in process 2085193
2024-04-21 09:03:06,547 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:06,548 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-04-21 09:03:06,550 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:06,551 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-04-20
2024-04-21 09:03:06,568 INFO [collect_river_data] running in process 2085197
2024-04-21 09:03:06,570 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:06,571 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-04-21 09:03:06,572 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:06,573 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-04-20
2024-04-21 09:03:06,578 INFO [collect_river_data] running in process 2085198
2024-04-21 09:03:06,579 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:06,580 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-21 09:03:06,581 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:06,582 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-04-20
2024-04-21 09:03:06,592 INFO [collect_river_data] running in process 2085195
2024-04-21 09:03:06,593 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:06,594 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-04-21 09:03:06,595 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:06,596 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-04-20
2024-04-21 09:03:06,742 INFO [crop_gribs] running in process 2085203
2024-04-21 09:03:06,743 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:06,744 INFO [crop_gribs] publishing log messages to tcp://*:5572
2024-04-21 09:03:06,745 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:06,746 INFO [crop_gribs] cropping 2024-04-21 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-04-21 09:03:06,751 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-04-20&endDT=2024-04-20 "HTTP/1.1 200 OK"
2024-04-21 09:03:06,757 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240421/18/
2024-04-21 09:03:06,770 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-04-21 09:03:06,772 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-04-20 completed
2024-04-21 09:03:06,775 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 09:03:06,922 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-04-20&endDT=2024-04-20 "HTTP/1.1 200 OK"
2024-04-21 09:03:06,931 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-04-21 09:03:06,933 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-04-20 completed
2024-04-21 09:03:06,934 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-04-20&endDT=2024-04-20 "HTTP/1.1 200 OK"
2024-04-21 09:03:06,935 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 09:03:06,938 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-04-21 09:03:06,938 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-04-20 completed
2024-04-21 09:03:06,944 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-04-20&endDT=2024-04-20 "HTTP/1.1 200 OK"
2024-04-21 09:03:06,949 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 09:03:06,954 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-04-20 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-04-21 09:03:06,955 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-04-20 completed
2024-04-21 09:03:06,958 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-21 09:03:07,640 INFO [collect_NeahBay_ssh] 2024-04-21T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-04-21 09:03:07,642 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-21 09:03:07,652 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 09:03:09,130 INFO [make_turbidity_file] 2024-04-21 Fraser River turbidity file creation complete
2024-04-21 09:03:09,132 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-04-21 09:03:10,214 INFO [make_ssh_files] running in process 2085356
2024-04-21 09:03:10,215 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:10,216 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2024-04-21 09:03:10,217 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:10,218 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-04-21 NOAA Neah Bay 06Z observation and forecast values
2024-04-21 09:03:14,638 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc
2024-04-21 09:03:14,800 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d18.nc
2024-04-21 09:03:14,963 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d19.nc
2024-04-21 09:03:15,122 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d20.nc
2024-04-21 09:03:15,313 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d21.nc
2024-04-21 09:03:15,477 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d22.nc
2024-04-21 09:03:15,640 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d23.nc
2024-04-21 09:03:15,802 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d24.nc
2024-04-21 09:03:15,931 INFO [make_ssh_files] sea surface height boundary file for 2024-04-21 nowcast run created
2024-04-21 09:03:15,934 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-21 09:03:15,945 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-21 09:03:15,949 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-21 09:03:16,969 INFO [make_runoff_file] running in process 2085390
2024-04-21 09:03:16,970 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:16,971 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2024-04-21 09:03:16,972 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:17,009 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-21 09:03:17,137 INFO [make_runoff_file] 2024-04-21 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-21 09:03:17,139 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-21 09:03:17,652 INFO [make_v202111_runoff_file] running in process 2085388
2024-04-21 09:03:17,653 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:03:17,654 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5563
2024-04-21 09:03:17,655 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:03:17,657 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-20
2024-04-21 09:03:20,270 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-20: /results/forcing/rivers/R202108Dailies_y2024m04d20.nc
2024-04-21 09:03:20,270 INFO [make_v202111_runoff_file] 2024-04-20 runoff file creation completed
2024-04-21 09:03:20,273 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-21 09:12:19,879 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240421/12/
2024-04-21 09:12:19,880 INFO [crop_gribs] 2024-04-21 12 GRIBs cropping complete
2024-04-21 09:12:19,882 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-21 09:12:20,089 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-04-21 09:12:22,169 INFO [grib_to_netcdf] running in process 2085449
2024-04-21 09:12:22,170 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:12:22,171 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2024-04-21 09:12:22,173 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:12:22,174 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-21 nowcast and forecast runs
2024-04-21 09:12:36,449 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m04d21.nc
2024-04-21 09:12:47,120 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d22.nc
2024-04-21 09:12:53,245 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d23.nc
2024-04-21 09:12:53,261 INFO [grib_to_netcdf] 2024-04-21 NEMO-atmos forcing file for nowcast+ created
2024-04-21 09:12:53,264 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-21 09:16:04,503 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 09:16:04,719 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-12-21'], host='localhost')
2024-04-21 09:16:04,726 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 09:16:04,731 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 09:16:05,772 INFO [download_results] running in process 2085566
2024-04-21 09:16:05,773 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:16:05,774 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-21 09:16:05,776 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:16:05,777 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21dec19 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 09:16:09,794 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 09:16:54,351 INFO [download_results] hindcast 2019-12-21 results files from optimum-hindcast downloaded
2024-04-21 09:16:54,366 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 09:16:54,376 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-12-21'], host='localhost')
2024-04-21 09:16:55,375 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 09:32:55,281 INFO [download_live_ocean] termination signal (SIGTERM) received; shutting down
2024-04-21 09:36:04,166 INFO [make_live_ocean_files] running in process 2085813
2024-04-21 09:36:04,167 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:36:04,169 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2024-04-21 09:36:04,170 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:36:04,171 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-04-21 Live Ocean run
2024-04-21 09:36:50,582 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2024-04-21 09:36:50,582 INFO [make_live_ocean_files] 2024-04-21 Live Ocean western boundary conditions files created
2024-04-21 09:36:50,585 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-04-21 09:36:50,594 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-04-21 09:36:50,597 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-04-21 09:36:50,601 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-04-21 09:36:50,605 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-04-21 09:36:50,609 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-04-21 09:36:51,429 INFO [ping_erddap] running in process 2085871
2024-04-21 09:36:51,430 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:36:51,431 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-21 09:36:51,432 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:36:51,434 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-04-21 09:36:51,436 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-21 09:36:51,568 INFO [upload_forcing] running in process 2085875
2024-04-21 09:36:51,569 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:36:51,570 INFO [upload_forcing] publishing log messages to tcp://*:5563
2024-04-21 09:36:51,571 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:36:51,572 INFO [upload_forcing] running in process 2085873
2024-04-21 09:36:51,573 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:36:51,575 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-21 09:36:51,576 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:36:51,600 INFO [upload_forcing] running in process 2085876
2024-04-21 09:36:51,601 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:36:51,602 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-21 09:36:51,603 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:36:51,639 INFO [upload_forcing] running in process 2085874
2024-04-21 09:36:51,640 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:36:51,641 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-21 09:36:51,642 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:36:53,940 INFO [upload_forcing] nowcast+ 2024-04-21 forcing files upload to optimum-hindcast completed
2024-04-21 09:36:53,944 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 09:36:54,260 INFO [upload_forcing] nowcast+ 2024-04-21 forcing files upload to orcinus-nowcast-agrif completed
2024-04-21 09:36:54,263 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 09:37:01,963 INFO [upload_forcing] nowcast+ 2024-04-21 forcing files upload to robot.graham completed
2024-04-21 09:37:01,968 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 09:37:04,472 INFO [upload_forcing] nowcast+ 2024-04-21 forcing files upload to arbutus.cloud-nowcast completed
2024-04-21 09:37:04,477 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 09:37:04,495 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 09:37:05,430 INFO [make_forcing_links] running in process 2085945
2024-04-21 09:37:05,431 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:37:05,432 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-04-21 09:37:05,433 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:37:08,384 INFO [make_forcing_links] nowcast+ 2024-04-21 forcing file links on arbutus.cloud-nowcast created
2024-04-21 09:37:08,388 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-21 09:37:08,405 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 09:37:10,444 INFO [run_NEMO] running in process 29907
2024-04-21 09:37:10,445 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:37:10,446 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-21 09:37:10,511 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:37:11,399 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/21apr24nowcast_2024-04-21T093710.491979-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-21 09:37:11,419 INFO [run_NEMO] nowcast NEMO run for 2024-04-21 on arbutus.cloud-nowcast started
2024-04-21 09:37:11,423 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-21 09:37:11,441 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-21 09:37:13,381 INFO [watch_NEMO] running in process 30161
2024-04-21 09:37:13,382 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 09:37:13,384 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-21 09:37:13,385 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-21 09:37:13,448 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-21 09:42:13,449 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17602273 = 2024-04-21 04:48:00 UTC, 20.0% complete
2024-04-21 09:47:13,450 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17602754 = 2024-04-21 10:08:40 UTC, 42.3% complete
2024-04-21 09:52:13,451 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17603213 = 2024-04-21 15:14:40 UTC, 63.5% complete
2024-04-21 09:57:13,452 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17603674 = 2024-04-21 20:22:00 UTC, 84.9% complete
2024-04-21 10:02:13,453 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17604000 = 2024-04-21 23:59:20 UTC, 100.0% complete
2024-04-21 10:07:13,456 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-04-21 10:07:13,460 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-21 10:07:13,673 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:07:13,680 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:07:14,693 INFO [download_results] running in process 2086212
2024-04-21 10:07:14,694 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:07:14,696 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-21 10:07:14,697 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:07:14,698 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/21apr24 to /results/SalishSea/nowcast-blue.202111
2024-04-21 10:07:14,700 INFO [make_forcing_links] running in process 2086211
2024-04-21 10:07:14,701 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:07:14,702 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2024-04-21 10:07:14,703 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:07:15,721 INFO [make_forcing_links] ssh 2024-04-21 forcing file links on arbutus.cloud-nowcast created
2024-04-21 10:07:15,726 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-21 10:07:15,745 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 10:07:18,179 INFO [run_NEMO] running in process 31625
2024-04-21 10:07:18,181 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:07:18,182 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-21 10:07:18,183 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:07:19,258 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/21apr24forecast_2024-04-21T100718.355070-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-21 10:07:19,278 INFO [run_NEMO] forecast NEMO run for 2024-04-21 on arbutus.cloud-nowcast started
2024-04-21 10:07:19,283 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-21 10:07:19,298 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-21 10:07:22,109 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-21 10:12:22,369 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17604444 = 2024-04-22 04:55:20 UTC, 13.7% complete
2024-04-21 10:17:22,697 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17604950 = 2024-04-22 10:32:40 UTC, 29.3% complete
2024-04-21 10:22:22,374 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17605454 = 2024-04-22 16:08:40 UTC, 44.9% complete
2024-04-21 10:27:19,287 INFO [download_results] nowcast 2024-04-21 results files from arbutus.cloud-nowcast downloaded
2024-04-21 10:27:19,292 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 10:27:19,308 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:27:19,313 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-04-20'], host='localhost')
2024-04-21 10:27:19,317 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:27:21,061 INFO [make_CHS_currents_file] running in process 2086294
2024-04-21 10:27:21,063 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:27:21,063 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-04-21 10:27:21,064 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:27:22,058 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17605905 = 2024-04-22 21:09:20 UTC, 58.8% complete
2024-04-21 10:27:22,384 INFO [make_plots] running in process 2086291
2024-04-21 10:27:22,385 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:27:22,387 INFO [make_plots] publishing log messages to tcp://*:5563
2024-04-21 10:27:22,388 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:27:22,394 INFO [make_plots] running in process 2086293
2024-04-21 10:27:22,395 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:27:22,396 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-21 10:27:22,397 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:27:23,061 INFO [make_plots] preparing render list for 2024-04-20 NEMO nowcast-blue comparison figures
2024-04-21 10:27:23,062 INFO [make_plots] preparing render list for 2024-04-21 NEMO nowcast-blue research figures
2024-04-21 10:27:24,516 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-blue research render list
2024-04-21 10:27:25,336 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-blue research render list
2024-04-21 10:27:25,338 INFO [make_plots] added Currents_sections_and_surface figure to 2024-04-21 NEMO nowcast-blue research render list
2024-04-21 10:27:25,339 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-04-21 NEMO nowcast-blue research render list
2024-04-21 10:27:25,340 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-04-21 NEMO nowcast-blue research render list
2024-04-21 10:27:25,341 INFO [make_plots] starting to render nemo nowcast research 21apr24 figures
2024-04-21 10:27:42,141 INFO [make_plots] added SH_wind figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,144 INFO [make_plots] added Compare_VENUS_East figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,145 INFO [make_plots] added Compare_VENUS_Central figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,147 INFO [make_plots] added Central_ADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,147 INFO [make_plots] added Central_depavADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,148 INFO [make_plots] added Central_timeavADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,150 INFO [make_plots] added East_ADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,151 INFO [make_plots] added East_depavADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,152 INFO [make_plots] added East_timeavADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,153 INFO [make_plots] added ddl_ADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,154 INFO [make_plots] added ddl_depavADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,155 INFO [make_plots] added ddl_timeavADCP figure to 2024-04-20 NEMO nowcast-blue comparison render list
2024-04-21 10:27:42,156 INFO [make_plots] starting to render nemo nowcast comparison 20apr24 figures
2024-04-21 10:27:54,791 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-21 for nowcast
2024-04-21 10:27:54,794 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-21 10:28:42,345 INFO [make_plots] finished rendering nemo nowcast research 21apr24 figures
2024-04-21 10:28:42,348 INFO [make_plots] nemo research plots for 2024-04-21 nowcast completed
2024-04-21 10:28:42,368 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-21 10:29:39,388 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-21 10:29:39,874 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-21 10:29:40,360 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-21 10:29:40,850 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-21 10:29:41,337 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-21 10:29:41,828 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-21 10:29:42,322 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-21 10:29:42,803 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-21 10:29:43,283 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-21 10:29:43,284 INFO [make_plots] finished rendering nemo nowcast comparison 20apr24 figures
2024-04-21 10:29:43,390 INFO [make_plots] nemo comparison plots for 2024-04-20 nowcast completed
2024-04-21 10:29:43,394 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-21 10:32:22,062 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17606393 = 2024-04-23 02:34:40 UTC, 73.8% complete
2024-04-21 10:37:22,060 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17606871 = 2024-04-23 07:53:20 UTC, 88.6% complete
2024-04-21 10:42:22,063 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17607240 = 2024-04-23 11:59:20 UTC, 100.0% complete
2024-04-21 10:47:22,069 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-04-21 10:47:22,072 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-21 10:47:22,289 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 10:47:22,296 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 10:47:22,300 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:47:22,305 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:47:22,310 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:47:22,315 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:47:22,320 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:47:23,554 INFO [download_results] running in process 2086439
2024-04-21 10:47:23,555 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:23,556 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-21 10:47:23,557 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:23,559 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/21apr24 to /results/SalishSea/forecast.202111
2024-04-21 10:47:23,565 INFO [upload_forcing] running in process 2086435
2024-04-21 10:47:23,566 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:23,567 INFO [upload_forcing] publishing log messages to tcp://*:5563
2024-04-21 10:47:23,568 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:23,574 INFO [upload_forcing] running in process 2086438
2024-04-21 10:47:23,576 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:23,576 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-21 10:47:23,578 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:23,579 INFO [upload_forcing] running in process 2086434
2024-04-21 10:47:23,580 INFO [upload_forcing] running in process 2086437
2024-04-21 10:47:23,581 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:23,581 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:23,582 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-21 10:47:23,583 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-21 10:47:23,584 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:23,585 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:24,254 INFO [upload_forcing] turbidity 2024-04-21 forcing files upload to optimum-hindcast completed
2024-04-21 10:47:24,267 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 10:47:24,376 INFO [upload_forcing] turbidity 2024-04-21 forcing files upload to orcinus-nowcast-agrif completed
2024-04-21 10:47:24,380 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 10:47:24,401 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:47:24,601 INFO [upload_forcing] turbidity 2024-04-21 forcing files upload to arbutus.cloud-nowcast completed
2024-04-21 10:47:24,605 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 10:47:24,625 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:47:25,742 INFO [make_forcing_links] running in process 2086470
2024-04-21 10:47:25,743 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:25,745 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2024-04-21 10:47:25,746 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:25,760 INFO [make_forcing_links] running in process 2086472
2024-04-21 10:47:25,761 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:25,763 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2024-04-21 10:47:25,764 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:26,885 INFO [make_forcing_links] nowcast-agrif 2024-04-21 forcing file links on orcinus-nowcast-agrif created
2024-04-21 10:47:26,888 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-21 10:47:26,894 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 10:47:26,912 INFO [make_forcing_links] nowcast-green 2024-04-21 forcing file links on arbutus.cloud-nowcast created
2024-04-21 10:47:26,916 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-21 10:47:26,938 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 10:47:27,523 INFO [upload_forcing] turbidity 2024-04-21 forcing files upload to robot.graham completed
2024-04-21 10:47:27,527 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-21 10:47:28,093 INFO [run_NEMO_agrif] running in process 2086486
2024-04-21 10:47:28,095 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:28,096 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5563
2024-04-21 10:47:28,097 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:28,802 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-21 10:47:29,124 INFO [run_NEMO] running in process 1658
2024-04-21 10:47:29,125 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:29,126 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-21 10:47:29,127 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:30,016 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/21apr24nowcast-green_2024-04-21T104729.160518-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-21 10:47:30,038 INFO [run_NEMO] nowcast-green NEMO run for 2024-04-21 on arbutus.cloud-nowcast started
2024-04-21 10:47:30,042 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-21 10:47:30,064 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-04-21 10:47:32,228 INFO [watch_NEMO] running in process 1915
2024-04-21 10:47:32,230 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:32,231 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-21 10:47:32,232 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:32,358 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-21 10:47:52,038 INFO [run_NEMO_agrif] job id for 21apr24nowcast-agrif: 11092006.orca2.ibb
2024-04-21 10:47:52,039 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-04-21 queued on orcinus-nowcast-agrif
2024-04-21 10:47:52,044 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-04-21 10:47:52,067 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11092006.orca2.ibb'], host='localhost')
2024-04-21 10:47:53,206 INFO [watch_NEMO_agrif] running in process 2086505
2024-04-21 10:47:53,207 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:47:53,209 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5563
2024-04-21 10:47:53,210 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:47:54,042 INFO [watch_NEMO_agrif] watching 21apr24nowcast-agrif job 11092006 on orcinus-nowcast-agrif
2024-04-21 10:47:54,279 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif job 11092006 is queued on orcinus-nowcast-agrif
2024-04-21 10:48:07,286 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-21 10:48:07,309 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 10:48:09,357 INFO [run_ww3] running in process 2016
2024-04-21 10:48:09,359 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:48:09,360 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-21 10:48:09,361 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:48:09,365 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-21T104809.339655-0700
2024-04-21 10:48:09,379 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-21T104809.339655-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-21 10:48:09,383 INFO [run_ww3] nowcast WaveWatch3 run for 2024-04-21 on arbutus.cloud-nowcast started
2024-04-21 10:48:09,434 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-21 10:48:09,482 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-21 10:48:11,580 INFO [watch_ww3] running in process 2113
2024-04-21 10:48:11,600 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:48:11,601 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-04-21 10:48:11,602 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:48:11,652 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-21 10:48:55,376 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-04-21 10:52:32,362 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17602006 = 2024-04-21 01:50:00 UTC, 7.6% complete
2024-04-21 10:53:11,657 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 72, 16.7% complete
2024-04-21 10:53:55,862 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7622832 = 2024-04-21 02:07:20 UTC, 8.8% complete
2024-04-21 10:56:16,065 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 10:56:16,298 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-12-26'], host='localhost')
2024-04-21 10:56:16,305 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 10:56:16,309 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 10:56:17,467 INFO [download_results] running in process 2086528
2024-04-21 10:56:17,468 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:56:17,470 INFO [download_results] publishing log messages to tcp://*:5567
2024-04-21 10:56:17,471 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:56:17,472 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26dec19 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 10:56:21,514 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 10:57:26,045 INFO [download_results] hindcast 2019-12-26 results files from optimum-hindcast downloaded
2024-04-21 10:57:26,061 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 10:57:26,082 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-12-26'], host='localhost')
2024-04-21 10:57:27,259 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 10:57:27,282 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-dec', 'robot.graham'], host='localhost')
2024-04-21 10:57:28,240 INFO [archive_tarball] running in process 2086573
2024-04-21 10:57:28,277 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 10:57:28,295 INFO [archive_tarball] publishing log messages to tcp://*:5566
2024-04-21 10:57:28,301 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-21 10:57:28,302 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-dec19.tar from /ocean/sallen/timbucktwo/oxygen/*dec19/
2024-04-21 10:57:32,352 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17602193 = 2024-04-21 03:54:40 UTC, 16.3% complete
2024-04-21 10:58:11,653 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 147, 34.0% complete
2024-04-21 10:58:56,316 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7623038 = 2024-04-21 04:24:40 UTC, 18.4% complete
2024-04-21 11:02:32,350 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17602379 = 2024-04-21 05:58:40 UTC, 24.9% complete
2024-04-21 11:03:11,656 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 228, 52.8% complete
2024-04-21 11:03:56,817 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7623245 = 2024-04-21 06:42:40 UTC, 28.0% complete
2024-04-21 11:04:13,630 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-dec19.index from /ocean/dlatorne/oxygen-dec19.tar
2024-04-21 11:04:13,733 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-dec19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-21 11:07:32,358 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17602564 = 2024-04-21 08:02:00 UTC, 33.5% complete
2024-04-21 11:08:11,653 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 306, 70.8% complete
2024-04-21 11:08:57,315 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7623451 = 2024-04-21 09:00:00 UTC, 37.5% complete
2024-04-21 11:12:32,361 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17602749 = 2024-04-21 10:05:20 UTC, 42.1% complete
2024-04-21 11:13:11,655 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 381, 88.2% complete
2024-04-21 11:13:57,801 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7623659 = 2024-04-21 11:18:40 UTC, 47.2% complete
2024-04-21 11:15:41,460 INFO [archive_tarball] hindcast *dec19 results files archived to robot.graham
2024-04-21 11:15:41,464 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-21 11:16:47,762 INFO [download_results] forecast 2024-04-21 results files from arbutus.cloud-nowcast downloaded
2024-04-21 11:16:47,766 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 11:16:47,789 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:16:51,134 INFO [make_CHS_currents_file] running in process 2086896
2024-04-21 11:16:51,135 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:16:51,136 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-04-21 11:16:51,137 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:17:32,351 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17602940 = 2024-04-21 12:12:40 UTC, 50.9% complete
2024-04-21 11:17:36,944 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-21 for forecast
2024-04-21 11:17:36,954 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-21 11:17:36,976 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:17:38,019 INFO [update_forecast_datasets] running in process 2086924
2024-04-21 11:17:38,020 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:17:38,021 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-04-21 11:17:38,022 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:17:38,023 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/21apr24 run
2024-04-21 11:17:38,329 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/21apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-21 11:17:38,330 INFO [update_forecast_datasets] nemo 2024-04-21 forecast rolling forecast datasets updated
2024-04-21 11:17:38,333 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-21 11:17:38,359 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-21 11:17:38,363 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:17:38,368 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:17:39,435 INFO [ping_erddap] running in process 2086929
2024-04-21 11:17:39,437 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:17:39,438 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-21 11:17:39,439 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:17:39,447 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-21 11:17:39,449 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-21 11:17:41,238 INFO [make_surface_current_tiles] running in process 2086932
2024-04-21 11:17:41,239 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:17:41,240 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2024-04-21 11:17:41,241 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:17:41,242 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-21 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/21apr24
2024-04-21 11:17:44,668 INFO [make_plots] running in process 2086931
2024-04-21 11:17:44,673 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:17:44,678 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-21 11:17:44,679 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:17:45,714 INFO [make_plots] preparing render list for 2024-04-21 NEMO forecast publish figures
2024-04-21 11:17:46,022 INFO [make_plots] added Website_thumbnail figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,023 INFO [make_plots] added Threshold_website figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,024 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,025 INFO [make_plots] added Boundary Bay figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,026 INFO [make_plots] added Campbell River figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,027 INFO [make_plots] added Cherry Point figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,028 INFO [make_plots] added Friday Harbor figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,028 INFO [make_plots] added Halfmoon Bay figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,029 INFO [make_plots] added Nanaimo figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,030 INFO [make_plots] added Neah Bay figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,031 INFO [make_plots] added New Westminster figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,031 INFO [make_plots] added Patricia Bay figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,032 INFO [make_plots] added Point Atkinson figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,033 INFO [make_plots] added Port Renfrew figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,034 INFO [make_plots] added Sand Heads figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,034 INFO [make_plots] added Sandy Cove figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,035 INFO [make_plots] added Squamish figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,035 INFO [make_plots] added Victoria figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,036 INFO [make_plots] added Woodwards Landing figure to 2024-04-21 NEMO forecast publish render list
2024-04-21 11:17:46,037 INFO [make_plots] starting to render nemo forecast publish 21apr24 figures
2024-04-21 11:18:11,649 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-21 11:18:11,650 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-21 11:18:11,889 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:18:11,914 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-21'], host='arbutus.cloud-nowcast')
2024-04-21 11:18:13,582 INFO [download_wwatch3_results] running in process 2087205
2024-04-21 11:18:13,583 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:18:13,584 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2024-04-21 11:18:13,585 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:18:13,868 INFO [run_ww3] running in process 4629
2024-04-21 11:18:13,869 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:18:13,870 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-21 11:18:13,871 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:18:13,874 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-21T111813.868125-0700
2024-04-21 11:18:13,876 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-21T111813.868125-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-21 11:18:13,894 INFO [run_ww3] forecast WaveWatch3 run for 2024-04-21 on arbutus.cloud-nowcast started
2024-04-21 11:18:13,954 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-21 11:18:14,012 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-21 11:18:16,024 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-21 11:18:58,288 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7623866 = 2024-04-21 13:36:40 UTC, 56.7% complete
2024-04-21 11:20:16,885 INFO [download_wwatch3_results] nowcast 2024-04-21 results files from arbutus.cloud-nowcast downloaded
2024-04-21 11:20:16,887 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-21 11:22:32,350 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17603126 = 2024-04-21 14:16:40 UTC, 59.5% complete
2024-04-21 11:23:16,023 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 66, 10.2% complete
2024-04-21 11:23:58,774 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7624073 = 2024-04-21 15:54:40 UTC, 66.3% complete
2024-04-21 11:27:32,350 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17603311 = 2024-04-21 16:20:00 UTC, 68.1% complete
2024-04-21 11:28:16,023 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 147, 22.7% complete
2024-04-21 11:28:59,225 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7624277 = 2024-04-21 18:10:40 UTC, 75.8% complete
2024-04-21 11:32:17,847 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/21apr24
2024-04-21 11:32:32,350 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17603493 = 2024-04-21 18:21:20 UTC, 76.5% complete
2024-04-21 11:33:16,024 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 225, 34.7% complete
2024-04-21 11:33:59,676 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7624485 = 2024-04-21 20:29:20 UTC, 85.4% complete
2024-04-21 11:36:27,522 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/21apr24
2024-04-21 11:36:27,531 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-21 forecast into /results/nowcast-sys/figures/surface_currents/forecast/21apr24
2024-04-21 11:36:27,532 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-21 forecast completed
2024-04-21 11:36:27,857 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-21 11:36:47,365 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-21 11:36:47,383 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-21 11:37:24,323 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:37:32,351 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17603677 = 2024-04-21 20:24:00 UTC, 85.0% complete
2024-04-21 11:38:16,024 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 306, 47.2% complete
2024-04-21 11:39:00,126 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7624693 = 2024-04-21 22:48:00 UTC, 95.0% complete
2024-04-21 11:39:18,016 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:39:18,440 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:39:55,254 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:41:07,877 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:41:45,501 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:42:23,985 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:42:32,352 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17603863 = 2024-04-21 22:28:00 UTC, 93.7% complete
2024-04-21 11:43:01,874 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:43:16,025 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 381, 58.8% complete
2024-04-21 11:43:39,389 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:44:00,563 INFO [watch_NEMO_agrif] 21apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7624800 = 2024-04-21 23:59:20 UTC, 100.0% complete
2024-04-21 11:44:16,664 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:44:56,170 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:44:56,608 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:45:34,753 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:46:11,874 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-16 00:00:00Z to 2024-04-23 11:55:00Z
2024-04-21 11:46:50,282 INFO [make_plots] finished rendering nemo forecast publish 21apr24 figures
2024-04-21 11:46:50,285 INFO [make_plots] nemo publish plots for 2024-04-21 forecast completed
2024-04-21 11:46:50,294 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-21 11:46:50,541 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:46:55,078 INFO [make_feeds] running in process 2089065
2024-04-21 11:46:55,079 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:46:55,081 INFO [make_feeds] publishing log messages to tcp://*:5562
2024-04-21 11:46:55,082 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:47:24,902 INFO [make_feeds] ATOM feeds for 2024-04-21 forecast run completed
2024-04-21 11:47:24,905 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-21 11:47:32,352 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17604000 = 2024-04-21 23:59:20 UTC, 100.0% complete
2024-04-21 11:48:16,025 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 462, 71.3% complete
2024-04-21 11:49:00,798 INFO [watch_NEMO_agrif] NEMO AGRIF run 11092006.orca2.ibb on orcinus-nowcast-agrif completed
2024-04-21 11:49:00,803 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-04-21 11:49:01,187 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:49:02,412 INFO [download_results] running in process 2089091
2024-04-21 11:49:02,414 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:49:02,415 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-21 11:49:02,416 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:49:02,417 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/21apr24 to /results/SalishSea/nowcast-agrif.201702
2024-04-21 11:49:25,337 INFO [download_results] nowcast-agrif 2024-04-21 results files from orcinus-nowcast-agrif downloaded
2024-04-21 11:49:25,347 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 11:49:25,502 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:49:29,486 INFO [make_plots] running in process 2089099
2024-04-21 11:49:29,487 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:49:29,489 INFO [make_plots] publishing log messages to tcp://*:5562
2024-04-21 11:49:29,490 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:49:30,092 INFO [make_plots] preparing render list for 2024-04-21 NEMO nowcast-agrif research figures
2024-04-21 11:49:30,093 INFO [make_plots] added baynes_sound_surface figure to 2024-04-21 NEMO nowcast-agrif research render list
2024-04-21 11:49:30,094 INFO [make_plots] starting to render nemo nowcast-agrif research 21apr24 figures
2024-04-21 11:49:41,421 INFO [make_plots] finished rendering nemo nowcast-agrif research 21apr24 figures
2024-04-21 11:49:41,423 INFO [make_plots] nemo research plots for 2024-04-21 nowcast-agrif completed
2024-04-21 11:49:41,426 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-21 11:52:32,359 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-04-21 11:52:32,363 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-21 11:52:32,740 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 11:52:34,085 INFO [download_results] running in process 2089140
2024-04-21 11:52:34,086 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 11:52:34,087 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-21 11:52:34,088 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 11:52:34,089 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/21apr24 to /results2/SalishSea/nowcast-green.202111
2024-04-21 11:53:16,043 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 540, 83.3% complete
2024-04-21 11:58:16,035 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 615, 94.9% complete
2024-04-21 12:03:16,034 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-21 12:03:16,056 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-21 12:03:16,465 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 12:03:17,699 INFO [download_wwatch3_results] running in process 2089303
2024-04-21 12:03:17,700 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:03:17,701 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2024-04-21 12:03:17,702 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:08:49,719 INFO [download_wwatch3_results] forecast 2024-04-21 results files from arbutus.cloud-nowcast downloaded
2024-04-21 12:08:49,723 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-21 12:08:49,944 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 12:08:51,114 INFO [update_forecast_datasets] running in process 2089434
2024-04-21 12:08:51,115 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:08:51,116 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2024-04-21 12:08:51,117 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:08:51,119 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/21apr24 run
2024-04-21 12:08:51,120 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/21apr24 run
2024-04-21 12:08:51,395 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/21apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-21 12:08:51,397 INFO [update_forecast_datasets] wwatch3 2024-04-21 forecast rolling forecast datasets updated
2024-04-21 12:08:51,399 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-21 12:08:51,579 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-21 12:08:52,616 INFO [ping_erddap] running in process 2089439
2024-04-21 12:08:52,618 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:08:52,618 INFO [ping_erddap] publishing log messages to tcp://*:5563
2024-04-21 12:08:52,620 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:08:52,621 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-21 12:08:52,623 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-21 12:08:52,808 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 12:08:57,535 INFO [make_plots] running in process 2089444
2024-04-21 12:08:57,536 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:08:57,537 INFO [make_plots] publishing log messages to tcp://*:5563
2024-04-21 12:08:57,538 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:08:57,539 INFO [make_plots] preparing render list for 2024-04-21 WaveWatch3 forecast2 publish figures
2024-04-21 12:08:57,540 INFO [make_plots] added Halibut Bank figure to 2024-04-21 WaveWatch3 forecast2 publish render list
2024-04-21 12:08:57,541 INFO [make_plots] added Sentry Shoal figure to 2024-04-21 WaveWatch3 forecast2 publish render list
2024-04-21 12:08:57,542 INFO [make_plots] starting to render wwatch3 forecast2 publish 21apr24 figures
2024-04-21 12:08:57,859 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-21 12:09:06,990 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-21 12:09:13,966 INFO [make_plots] finished rendering wwatch3 forecast2 publish 21apr24 figures
2024-04-21 12:09:13,966 INFO [make_plots] wwatch3 publish plots for 2024-04-21 forecast2 completed
2024-04-21 12:09:13,970 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-21 12:21:26,200 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 12:21:26,696 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-01-01'], host='localhost')
2024-04-21 12:21:26,702 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 12:21:26,707 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 12:21:28,094 INFO [download_results] running in process 2089504
2024-04-21 12:21:28,095 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:21:28,096 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-21 12:21:28,097 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:21:28,098 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01jan20 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 12:21:32,420 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 12:22:28,598 INFO [download_results] hindcast 2020-01-01 results files from optimum-hindcast downloaded
2024-04-21 12:22:28,614 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 12:22:28,796 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-01-01'], host='localhost')
2024-04-21 12:22:29,789 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 12:29:33,345 INFO [download_results] nowcast-green 2024-04-21 results files from arbutus.cloud-nowcast downloaded
2024-04-21 12:29:33,359 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 12:29:33,648 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 12:29:33,654 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-04-21 12:29:33,659 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 12:29:33,664 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 12:29:33,667 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-04-21'], host='localhost')
2024-04-21 12:29:37,272 INFO [ping_erddap] running in process 2089600
2024-04-21 12:29:37,332 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:29:37,514 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-21 12:29:37,529 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:29:37,651 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-04-21 12:29:37,653 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-21 12:29:46,370 INFO [make_averaged_dataset] running in process 2089601
2024-04-21 12:29:46,372 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:29:46,416 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2024-04-21 12:29:46,450 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:29:46,488 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Apr-2024 physics
2024-04-21 12:29:46,508 INFO [make_averaged_dataset] running in process 2089602
2024-04-21 12:29:46,510 INFO [make_averaged_dataset] running in process 2089603
2024-04-21 12:29:46,511 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:29:46,512 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:29:46,514 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2024-04-21 12:29:46,515 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2024-04-21 12:29:46,536 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:29:46,553 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:29:46,554 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Apr-2024 chemistry
2024-04-21 12:29:46,555 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Apr-2024 biology
2024-04-21 12:29:53,034 INFO [make_plots] running in process 2089598
2024-04-21 12:29:53,035 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 12:29:53,037 INFO [make_plots] publishing log messages to tcp://*:5567
2024-04-21 12:29:53,038 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-21 12:29:53,877 INFO [make_plots] preparing render list for 2024-04-21 NEMO nowcast-green research figures
2024-04-21 12:29:55,747 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:29:56,566 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:29:57,449 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:29:58,290 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:29:59,128 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:29:59,900 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:00,697 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:01,602 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:02,517 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:03,367 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:04,234 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:04,957 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:05,336 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:05,336 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:05,337 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:05,338 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-04-21 NEMO nowcast-green research render list
2024-04-21 12:30:05,339 INFO [make_plots] starting to render nemo nowcast-green research 21apr24 figures
2024-04-21 12:31:03,827 INFO [make_averaged_dataset] day-averaged dataset for 21-Apr-2024 physics created
2024-04-21 12:31:03,830 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-21 12:31:56,648 INFO [make_averaged_dataset] day-averaged dataset for 21-Apr-2024 chemistry created
2024-04-21 12:31:56,650 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-21 12:32:14,057 INFO [make_averaged_dataset] day-averaged dataset for 21-Apr-2024 biology created
2024-04-21 12:32:14,061 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-21 13:46:36,742 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 13:46:37,161 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-01-06'], host='localhost')
2024-04-21 13:46:37,167 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 13:46:37,171 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 13:46:39,347 INFO [download_results] running in process 2090334
2024-04-21 13:46:39,349 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 13:46:39,350 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-21 13:46:39,351 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 13:46:39,352 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06jan20 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 13:46:43,573 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 13:47:54,933 INFO [download_results] hindcast 2020-01-06 results files from optimum-hindcast downloaded
2024-04-21 13:47:55,027 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 13:47:55,201 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-01-06'], host='localhost')
2024-04-21 13:47:56,755 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 13:59:36,312 INFO [make_plots] finished rendering nemo nowcast-green research 21apr24 figures
2024-04-21 13:59:36,491 INFO [make_plots] nemo research plots for 2024-04-21 nowcast-green completed
2024-04-21 13:59:36,562 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-21 15:08:40,479 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240421/18/
2024-04-21 15:08:40,480 INFO [collect_weather] 2024-04-21 2.5km weather forecast 18 collection complete
2024-04-21 15:08:40,483 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-21 15:08:40,892 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-04-21 15:08:40,898 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-04-21 15:08:40,902 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-04-21 15:08:40,907 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-04-22'], host='localhost')
2024-04-21 15:08:42,429 INFO [download_weather] running in process 2090913
2024-04-21 15:08:42,430 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 15:08:42,431 INFO [download_weather] publishing log messages to tcp://*:5563
2024-04-21 15:08:42,436 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-21 15:08:42,437 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240421
2024-04-21 15:08:42,439 INFO [collect_weather] running in process 2090916
2024-04-21 15:08:42,441 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 15:08:42,442 INFO [collect_weather] publishing log messages to tcp://*:5564
2024-04-21 15:08:42,443 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-21 15:08:42,444 INFO [download_weather] running in process 2090915
2024-04-21 15:08:42,445 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 15:08:42,445 INFO [download_weather] publishing log messages to tcp://*:5565
2024-04-21 15:08:42,446 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-21 15:08:42,447 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240421
2024-04-21 15:08:42,512 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-04-21 15:08:45,393 INFO [crop_gribs] running in process 2090917
2024-04-21 15:08:45,394 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 15:08:45,395 INFO [crop_gribs] publishing log messages to tcp://*:5566
2024-04-21 15:08:45,396 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-21 15:08:45,397 INFO [crop_gribs] cropping 2024-04-22 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-04-21 15:08:45,406 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240422/00/
2024-04-21 15:11:47,952 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 15:11:48,350 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-01-11'], host='localhost')
2024-04-21 15:11:48,355 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 15:11:48,358 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 15:11:49,756 INFO [download_results] running in process 2090960
2024-04-21 15:11:49,757 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 15:11:49,758 INFO [download_results] publishing log messages to tcp://*:5568
2024-04-21 15:11:49,760 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 15:11:49,761 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11jan20 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 15:11:53,745 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 15:12:53,314 INFO [download_results] hindcast 2020-01-11 results files from optimum-hindcast downloaded
2024-04-21 15:12:53,329 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 15:12:53,569 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-01-11'], host='localhost')
2024-04-21 15:12:54,717 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 15:13:08,351 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240421/18/
2024-04-21 15:13:08,352 INFO [crop_gribs] 2024-04-21 18 GRIBs cropping complete
2024-04-21 15:13:08,354 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-21 15:16:49,158 INFO [download_weather] 2024-04-21 1km weather forecast 12 downloads complete
2024-04-21 15:16:49,179 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-21 15:16:58,082 INFO [download_weather] 2024-04-21 1km weather forecast 00 downloads complete
2024-04-21 15:16:58,106 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-21 16:41:58,445 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 16:41:58,917 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-01-16'], host='localhost')
2024-04-21 16:41:58,924 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 16:41:58,930 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 16:42:00,623 INFO [download_results] running in process 2091350
2024-04-21 16:42:00,624 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 16:42:00,625 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-21 16:42:00,626 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 16:42:00,627 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16jan20 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 16:42:04,815 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 16:43:03,512 INFO [download_results] hindcast 2020-01-16 results files from optimum-hindcast downloaded
2024-04-21 16:43:03,526 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 16:43:03,740 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-01-16'], host='localhost')
2024-04-21 16:43:04,982 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 18:07:09,372 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 18:07:09,811 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-01-21'], host='localhost')
2024-04-21 18:07:09,817 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 18:07:09,823 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 18:07:11,470 INFO [download_results] running in process 2091953
2024-04-21 18:07:11,472 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 18:07:11,473 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-21 18:07:11,474 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 18:07:11,475 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21jan20 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 18:07:15,502 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 18:08:19,509 INFO [download_results] hindcast 2020-01-21 results files from optimum-hindcast downloaded
2024-04-21 18:08:19,525 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 18:08:19,755 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-01-21'], host='localhost')
2024-04-21 18:08:21,032 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 19:47:21,697 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 19:47:22,148 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-01-26'], host='localhost')
2024-04-21 19:47:22,154 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 19:47:22,160 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 19:47:23,781 INFO [download_results] running in process 2092220
2024-04-21 19:47:23,782 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 19:47:23,783 INFO [download_results] publishing log messages to tcp://*:5563
2024-04-21 19:47:23,784 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 19:47:23,785 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jan20 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 19:47:28,236 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 19:48:36,465 INFO [download_results] hindcast 2020-01-26 results files from optimum-hindcast downloaded
2024-04-21 19:48:36,483 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 19:48:36,727 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-01-26'], host='localhost')
2024-04-21 19:48:38,083 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 19:48:38,313 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2020-jan', 'robot.graham'], host='localhost')
2024-04-21 19:48:39,306 INFO [archive_tarball] running in process 2092268
2024-04-21 19:48:39,331 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 19:48:39,349 INFO [archive_tarball] publishing log messages to tcp://*:5563
2024-04-21 19:48:39,367 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-21 19:48:39,372 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jan20.tar from /ocean/sallen/timbucktwo/oxygen/*jan20/
2024-04-21 19:56:32,993 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jan20.index from /ocean/dlatorne/oxygen-jan20.tar
2024-04-21 19:56:33,101 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jan20.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-21 20:05:54,597 INFO [archive_tarball] hindcast *jan20 results files archived to robot.graham
2024-04-21 20:05:54,620 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-21 21:09:35,127 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20240422/00/
2024-04-21 21:09:35,147 INFO [collect_weather] 2024-04-22 2.5km weather forecast 00 collection complete
2024-04-21 21:09:35,183 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-21 21:09:35,672 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2024-04-21 21:09:35,678 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2024-04-22'], host='localhost')
2024-04-21 21:09:37,267 INFO [collect_weather] running in process 2092551
2024-04-21 21:09:37,269 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 21:09:37,270 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-04-21 21:09:37,270 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-21 21:09:37,293 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2024-04-21 21:09:39,656 INFO [crop_gribs] running in process 2092553
2024-04-21 21:09:39,657 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 21:09:39,659 INFO [crop_gribs] publishing log messages to tcp://*:5564
2024-04-21 21:09:39,660 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-21 21:09:39,661 INFO [crop_gribs] cropping 2024-04-22 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2024-04-21 21:09:39,669 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240422/06/
2024-04-21 21:12:32,315 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 21:12:32,740 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-02-01'], host='localhost')
2024-04-21 21:12:32,747 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 21:12:32,751 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 21:12:34,152 INFO [download_results] running in process 2092702
2024-04-21 21:12:34,154 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 21:12:34,155 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-21 21:12:34,156 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 21:12:34,157 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01feb20 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 21:12:38,217 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 21:13:36,918 INFO [download_results] hindcast 2020-02-01 results files from optimum-hindcast downloaded
2024-04-21 21:13:36,931 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 21:13:37,150 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-02-01'], host='localhost')
2024-04-21 21:13:38,380 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-21 21:13:42,147 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240422/00/
2024-04-21 21:13:42,148 INFO [crop_gribs] 2024-04-22 00 GRIBs cropping complete
2024-04-21 21:13:42,150 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-21 22:37:42,766 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-21 22:37:43,169 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-02-06'], host='localhost')
2024-04-21 22:37:43,176 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 22:37:43,181 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-21 22:37:44,787 INFO [download_results] running in process 2093364
2024-04-21 22:37:44,788 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-21 22:37:44,789 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-21 22:37:44,790 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-21 22:37:44,791 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06feb20 to /ocean/sallen/timbucktwo/oxygen
2024-04-21 22:37:48,869 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-21 22:38:47,574 INFO [download_results] hindcast 2020-02-06 results files from optimum-hindcast downloaded
2024-04-21 22:38:47,603 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-21 22:38:47,895 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-02-06'], host='localhost')
2024-04-21 22:38:49,167 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 00:07:53,612 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 00:07:54,026 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-02-11'], host='localhost')
2024-04-22 00:07:54,044 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 00:07:54,050 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 00:07:55,455 INFO [download_results] running in process 2094059
2024-04-22 00:07:55,456 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 00:07:55,457 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-22 00:07:55,458 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 00:07:55,459 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11feb20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 00:07:59,587 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 00:08:56,917 INFO [download_results] hindcast 2020-02-11 results files from optimum-hindcast downloaded
2024-04-22 00:08:56,933 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 00:08:57,149 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-02-11'], host='localhost')
2024-04-22 00:08:58,410 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 01:33:05,150 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 01:33:05,576 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-02-16'], host='localhost')
2024-04-22 01:33:05,582 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 01:33:05,588 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 01:33:07,198 INFO [download_results] running in process 2094511
2024-04-22 01:33:07,199 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 01:33:07,200 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-22 01:33:07,201 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 01:33:07,202 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16feb20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 01:33:11,221 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 01:34:09,777 INFO [download_results] hindcast 2020-02-16 results files from optimum-hindcast downloaded
2024-04-22 01:34:09,792 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 01:34:10,010 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-02-16'], host='localhost')
2024-04-22 01:34:11,202 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 02:58:16,000 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 02:58:16,415 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-02-21'], host='localhost')
2024-04-22 02:58:16,421 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 02:58:16,426 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 02:58:17,636 INFO [download_results] running in process 2095600
2024-04-22 02:58:17,637 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 02:58:17,638 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-22 02:58:17,638 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 02:58:17,639 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21feb20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 02:58:21,685 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 02:59:06,353 INFO [download_results] hindcast 2020-02-21 results files from optimum-hindcast downloaded
2024-04-22 02:59:06,369 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 02:59:06,585 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-02-21'], host='localhost')
2024-04-22 02:59:07,637 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 03:02:21,919 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20240422/06/
2024-04-22 03:02:21,920 INFO [collect_weather] 2024-04-22 2.5km weather forecast 06 collection complete
2024-04-22 03:02:21,923 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-22 03:02:22,320 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,326 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,330 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,335 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,339 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,344 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,346 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,350 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,358 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,365 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,369 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,382 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,389 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,400 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2024-04-21'], host='localhost')
2024-04-22 03:02:22,425 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2024-04-22 03:02:22,433 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2024-04-22 03:02:22,452 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2024-04-22 03:02:22,464 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2024-04-22 03:02:22,486 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2024-04-22 03:02:22,502 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2024-04-22 03:02:24,405 INFO [collect_NeahBay_ssh] running in process 2095750
2024-04-22 03:02:24,407 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:24,408 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2024-04-22 03:02:24,409 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:24,409 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2024-04-22
2024-04-22 03:02:24,413 INFO [collect_weather] running in process 2095751
2024-04-22 03:02:24,413 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:24,415 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-04-22 03:02:24,415 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:24,426 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2024-04-22 03:02:25,986 INFO [collect_river_data] running in process 2095734
2024-04-22 03:02:25,990 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:25,996 INFO [collect_river_data] publishing log messages to tcp://*:5565
2024-04-22 03:02:26,003 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,005 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2024-04-21
2024-04-22 03:02:26,080 INFO [collect_river_data] running in process 2095732
2024-04-22 03:02:26,080 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,081 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-04-22 03:02:26,082 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,083 INFO [collect_river_data] Collecting ECCC Capilano river data for 2024-04-21
2024-04-22 03:02:26,099 INFO [collect_river_data] running in process 2095743
2024-04-22 03:02:26,100 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,101 INFO [collect_river_data] publishing log messages to tcp://*:5568
2024-04-22 03:02:26,103 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,106 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2024-04-21
2024-04-22 03:02:26,109 INFO [collect_river_data] running in process 2095738
2024-04-22 03:02:26,110 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,111 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-04-22 03:02:26,112 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,113 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2024-04-21
2024-04-22 03:02:26,133 INFO [collect_river_data] running in process 2095746
2024-04-22 03:02:26,134 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,135 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-22 03:02:26,136 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,137 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2024-04-21
2024-04-22 03:02:26,178 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2024-04-22 03:02:26,179 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2024-04-21 completed
2024-04-22 03:02:26,184 INFO [collect_river_data] running in process 2095739
2024-04-22 03:02:26,185 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2024-04-22 03:02:26,186 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:26,187 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,187 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2024-04-21 completed
2024-04-22 03:02:26,188 INFO [collect_river_data] publishing log messages to tcp://*:5571
2024-04-22 03:02:26,189 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,190 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2024-04-21
2024-04-22 03:02:26,191 INFO [collect_river_data] running in process 2095744
2024-04-22 03:02:26,192 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,193 INFO [collect_river_data] publishing log messages to tcp://*:5572
2024-04-22 03:02:26,194 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,196 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2024-04-21
2024-04-22 03:02:26,198 INFO [collect_river_data] running in process 2095740
2024-04-22 03:02:26,200 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,201 INFO [collect_river_data] publishing log messages to tcp://*:5573
2024-04-22 03:02:26,217 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,218 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2024-04-21
2024-04-22 03:02:26,219 INFO [collect_river_data] running in process 2095742
2024-04-22 03:02:26,221 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,221 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2024-04-22 03:02:26,224 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2024-04-21 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2024-04-22 03:02:26,225 INFO [collect_river_data] publishing log messages to tcp://*:5574
2024-04-22 03:02:26,226 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2024-04-21 completed
2024-04-22 03:02:26,227 INFO [collect_river_data] ECCC Capilano river data collection for 2024-04-21 completed
2024-04-22 03:02:26,228 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,230 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2024-04-21
2024-04-22 03:02:26,232 INFO [collect_river_data] running in process 2095737
2024-04-22 03:02:26,233 INFO [collect_river_data] running in process 2095741
2024-04-22 03:02:26,234 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,235 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,235 INFO [collect_river_data] publishing log messages to tcp://*:5575
2024-04-22 03:02:26,236 INFO [collect_river_data] publishing log messages to tcp://*:5576
2024-04-22 03:02:26,238 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,239 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,239 INFO [collect_river_data] Collecting ECCC Fraser river data for 2024-04-21
2024-04-22 03:02:26,240 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2024-04-21
2024-04-22 03:02:26,250 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2024-04-22 03:02:26,251 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2024-04-21 completed
2024-04-22 03:02:26,265 INFO [collect_river_data] running in process 2095736
2024-04-22 03:02:26,266 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,267 INFO [collect_river_data] publishing log messages to tcp://*:5577
2024-04-22 03:02:26,267 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,268 INFO [collect_river_data] Collecting ECCC Englishman river data for 2024-04-21
2024-04-22 03:02:26,270 INFO [collect_river_data] running in process 2095745
2024-04-22 03:02:26,270 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,273 INFO [collect_river_data] publishing log messages to tcp://*:5578
2024-04-22 03:02:26,274 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,275 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2024-04-21
2024-04-22 03:02:26,283 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2024-04-22 03:02:26,284 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2024-04-21 completed
2024-04-22 03:02:26,286 INFO [collect_river_data] running in process 2095735
2024-04-22 03:02:26,287 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,288 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2024-04-22 03:02:26,289 INFO [collect_river_data] publishing log messages to tcp://*:5579
2024-04-22 03:02:26,290 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2024-04-21 completed
2024-04-22 03:02:26,291 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,291 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2024-04-21
2024-04-22 03:02:26,309 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/RobertsCreek_flow
2024-04-22 03:02:26,310 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2024-04-21 completed
2024-04-22 03:02:26,315 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2024-04-22 03:02:26,315 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2024-04-21 completed
2024-04-22 03:02:26,327 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2024-04-22 03:02:26,329 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2024-04-21 completed
2024-04-22 03:02:26,331 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2024-04-21 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2024-04-22 03:02:26,332 INFO [collect_river_data] ECCC Fraser river data collection for 2024-04-21 completed
2024-04-22 03:02:26,363 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2024-04-22 03:02:26,365 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2024-04-21 completed
2024-04-22 03:02:26,379 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2024-04-21 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2024-04-22 03:02:26,380 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2024-04-21 completed
2024-04-22 03:02:26,435 INFO [collect_NeahBay_ssh] 2024-04-22T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2024-04-22 03:02:26,506 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:26,589 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2024-04-21 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2024-04-22 03:02:26,590 INFO [collect_river_data] ECCC Englishman river data collection for 2024-04-21 completed
2024-04-22 03:02:26,735 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:26,858 INFO [crop_gribs] running in process 2095752
2024-04-22 03:02:26,859 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:26,860 INFO [crop_gribs] publishing log messages to tcp://*:5565
2024-04-22 03:02:26,861 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:26,862 INFO [crop_gribs] cropping 2024-04-22 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2024-04-22 03:02:26,875 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240422/12/
2024-04-22 03:02:26,938 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:27,126 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:27,191 INFO [get_onc_ctd] running in process 2095747
2024-04-22 03:02:27,192 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:27,193 INFO [get_onc_ctd] publishing log messages to tcp://*:5569
2024-04-22 03:02:27,195 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:27,196 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2024-04-21
2024-04-22 03:02:27,198 INFO [get_onc_ferry] running in process 2095749
2024-04-22 03:02:27,199 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:27,200 INFO [get_onc_ferry] publishing log messages to tcp://*:5568
2024-04-22 03:02:27,201 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:27,202 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2024-04-21
2024-04-22 03:02:27,285 INFO [get_onc_ctd] running in process 2095748
2024-04-22 03:02:27,287 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:27,288 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2024-04-22 03:02:27,289 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:27,290 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2024-04-21
2024-04-22 03:02:27,318 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:27,505 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:27,710 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:27,896 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:28,081 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:28,277 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:28,461 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:28,645 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:28,844 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-22 03:02:29,028 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 03:02:29,035 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-22 03:02:32,380 INFO [make_ssh_files] running in process 2096158
2024-04-22 03:02:32,381 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:32,382 INFO [make_ssh_files] publishing log messages to tcp://*:5562
2024-04-22 03:02:32,383 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:32,384 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2024-04-22 NOAA Neah Bay 00Z observation and forecast values
2024-04-22 03:02:37,366 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc
2024-04-22 03:02:37,603 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d18.nc
2024-04-22 03:02:37,825 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d19.nc
2024-04-22 03:02:38,019 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d20.nc
2024-04-22 03:02:38,200 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d21.nc
2024-04-22 03:02:38,431 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d22.nc
2024-04-22 03:02:38,624 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d23.nc
2024-04-22 03:02:38,801 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d24.nc
2024-04-22 03:02:39,005 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d25.nc
2024-04-22 03:02:39,167 INFO [make_ssh_files] sea surface height boundary file for 2024-04-22 forecast2 run created
2024-04-22 03:02:39,171 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-22 03:02:39,381 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-22 03:02:39,386 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-22 03:02:40,536 INFO [make_runoff_file] running in process 2096193
2024-04-22 03:02:40,537 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:40,538 INFO [make_runoff_file] publishing log messages to tcp://*:5562
2024-04-22 03:02:40,539 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:40,606 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-22 03:02:40,704 INFO [make_runoff_file] 2024-04-22 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-22 03:02:40,706 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-22 03:02:41,177 INFO [make_v202111_runoff_file] running in process 2096191
2024-04-22 03:02:41,178 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:41,179 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5562
2024-04-22 03:02:41,180 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:41,181 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-21
2024-04-22 03:02:42,235 ERROR [make_v202111_runoff_file] no 2024-04-21 discharge obs for Skagit_MountVernon: patching
2024-04-22 03:02:42,784 ERROR [make_v202111_runoff_file] no 2024-04-21 discharge obs for Snohomish_Monroe: patching
2024-04-22 03:02:43,049 ERROR [make_v202111_runoff_file] no 2024-04-21 discharge obs for Nisqually_McKenna: patching
2024-04-22 03:02:43,168 ERROR [make_v202111_runoff_file] no 2024-04-21 discharge obs for Greenwater_Greenwater: patching
2024-04-22 03:02:44,558 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-21: /results/forcing/rivers/R202108Dailies_y2024m04d21.nc
2024-04-22 03:02:44,559 INFO [make_v202111_runoff_file] 2024-04-21 runoff file creation completed
2024-04-22 03:02:44,562 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-22 03:02:45,584 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2024-04-21
2024-04-22 03:02:47,132 INFO [get_onc_ctd] 2024-04-21 ONC SEVIP CTD T&S file created
2024-04-22 03:02:47,135 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2024-04-22 03:02:47,349 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2024-04-22 03:02:48,152 INFO [ping_erddap] running in process 2096233
2024-04-22 03:02:48,153 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:02:48,154 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-22 03:02:48,155 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:02:48,157 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2024-04-22 03:02:48,158 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-22 03:03:04,909 CRITICAL [get_onc_ctd] 2024-04-21 ONC SCVIP CTD T&S file creation failed
2024-04-22 03:03:10,937 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2024-04-21
2024-04-22 03:03:37,722 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2024-04-21
2024-04-22 03:04:07,758 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2024-04-21
2024-04-22 03:04:24,112 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2024-04-21
2024-04-22 03:04:41,272 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2024-04-21
2024-04-22 03:04:50,097 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2024-04-21
2024-04-22 03:04:58,880 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2024-04-21; substituting empty dataset
2024-04-22 03:04:58,880 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2024-04-21
2024-04-22 03:05:07,061 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-21; substituting empty dataset
2024-04-22 03:06:17,219 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2024-04-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-04-22 03:06:17,237 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2024-04-21 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2024-04-22 03:06:17,563 INFO [get_onc_ferry] 2024-04-21 ONC TWDP ferry data file created
2024-04-22 03:06:17,566 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2024-04-22 03:06:17,762 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2024-04-22 03:06:18,679 INFO [ping_erddap] running in process 2096267
2024-04-22 03:06:18,680 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:06:18,681 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-22 03:06:18,682 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:06:18,684 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2024-04-22 03:06:18,686 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-22 03:11:19,023 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240422/06/
2024-04-22 03:11:19,024 INFO [crop_gribs] 2024-04-22 06 GRIBs cropping complete
2024-04-22 03:11:19,026 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-22 03:11:19,424 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2024-04-22 03:11:21,299 INFO [grib_to_netcdf] running in process 2096292
2024-04-22 03:11:21,300 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:11:21,302 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2024-04-22 03:11:21,303 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:11:21,304 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-22 forecast2 run
2024-04-22 03:11:32,862 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d23.nc
2024-04-22 03:11:36,405 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d24.nc
2024-04-22 03:11:36,415 INFO [grib_to_netcdf] 2024-04-22 NEMO-atmos forcing file for forecast2 created
2024-04-22 03:11:36,418 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-22 03:11:36,588 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2024-04-22 03:11:36,590 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2024-04-22 03:11:36,596 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2024-04-22 03:11:36,601 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2024-04-22 03:11:37,588 INFO [upload_forcing] running in process 2096332
2024-04-22 03:11:37,589 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:11:37,590 INFO [upload_forcing] publishing log messages to tcp://*:5562
2024-04-22 03:11:37,591 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:11:37,596 INFO [upload_forcing] running in process 2096331
2024-04-22 03:11:37,598 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:11:37,599 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-04-22 03:11:37,600 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:11:37,613 INFO [upload_forcing] running in process 2096328
2024-04-22 03:11:37,614 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:11:37,615 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-22 03:11:37,616 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:11:37,621 INFO [upload_forcing] running in process 2096330
2024-04-22 03:11:37,622 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:11:37,623 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-22 03:11:37,624 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:11:39,132 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2024m04d21.nc
2024-04-22 03:11:39,207 INFO [upload_forcing] forecast2 2024-04-22 forcing files upload to optimum-hindcast completed
2024-04-22 03:11:39,210 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 03:11:39,215 INFO [upload_forcing] forecast2 2024-04-22 forcing files upload to orcinus-nowcast-agrif completed
2024-04-22 03:11:39,456 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 03:11:40,049 INFO [upload_forcing] forecast2 2024-04-22 forcing files upload to arbutus.cloud-nowcast completed
2024-04-22 03:11:40,053 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 03:11:40,272 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 03:11:41,346 INFO [make_forcing_links] running in process 2096360
2024-04-22 03:11:41,348 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:11:41,349 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2024-04-22 03:11:41,350 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:11:42,537 INFO [make_forcing_links] forecast2 2024-04-22 forcing file links on arbutus.cloud-nowcast created
2024-04-22 03:11:42,540 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-22 03:11:42,829 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 03:11:44,720 INFO [upload_forcing] forecast2 2024-04-22 forcing files upload to robot.graham completed
2024-04-22 03:11:44,725 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-22 03:11:44,806 INFO [run_NEMO] running in process 1822
2024-04-22 03:11:44,807 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:11:44,808 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-22 03:11:44,817 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:11:45,897 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/21apr24forecast2_2024-04-22T031144.997641-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-22 03:11:45,914 INFO [run_NEMO] forecast2 NEMO run for 2024-04-22 on arbutus.cloud-nowcast started
2024-04-22 03:11:45,918 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-22 03:11:46,141 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-22 03:11:48,055 INFO [watch_NEMO] running in process 2080
2024-04-22 03:11:48,056 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:11:48,057 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-22 03:11:48,058 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:11:48,127 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-22 03:16:48,127 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17606597 = 2024-04-23 04:50:40 UTC, 16.2% complete
2024-04-22 03:21:48,128 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17607074 = 2024-04-23 10:08:40 UTC, 33.8% complete
2024-04-22 03:26:48,128 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17607552 = 2024-04-23 15:27:20 UTC, 51.5% complete
2024-04-22 03:31:48,129 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17608014 = 2024-04-23 20:35:20 UTC, 68.7% complete
2024-04-22 03:36:48,131 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17608481 = 2024-04-24 01:46:40 UTC, 86.0% complete
2024-04-22 03:41:48,132 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 17608860 = 2024-04-24 05:59:20 UTC, 100.0% complete
2024-04-22 03:46:48,136 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2024-04-22 03:46:48,140 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-22 03:46:48,543 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 03:46:48,550 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 03:46:48,554 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-22 03:46:49,690 INFO [download_results] running in process 2096569
2024-04-22 03:46:49,691 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 03:46:49,693 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-22 03:46:49,694 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 03:46:49,695 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/21apr24 to /results/SalishSea/forecast2.202111
2024-04-22 03:46:55,025 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-22 03:48:44,081 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-22 03:48:44,299 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-22'], host='arbutus.cloud-nowcast')
2024-04-22 03:48:55,930 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-22 03:48:56,149 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-22 03:49:08,476 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-22 03:54:08,472 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 72, 13.3% complete
2024-04-22 03:59:08,476 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 153, 28.3% complete
2024-04-22 04:04:08,466 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 228, 42.2% complete
2024-04-22 04:08:29,649 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-22 04:08:30,073 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2020-02-26'], host='localhost')
2024-04-22 04:08:30,080 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 04:08:30,085 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-22 04:08:31,046 INFO [download_results] running in process 2096618
2024-04-22 04:08:31,047 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:08:31,048 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-22 04:08:31,050 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:08:31,051 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26feb20 to /ocean/sallen/timbucktwo/oxygen
2024-04-22 04:08:35,273 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-22 04:09:07,884 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 309, 57.2% complete
2024-04-22 04:09:13,347 INFO [download_results] hindcast 2020-02-26 results files from optimum-hindcast downloaded
2024-04-22 04:09:13,356 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 04:09:13,568 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2020-02-26'], host='localhost')
2024-04-22 04:09:14,557 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-22 04:09:14,773 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2020-feb', 'robot.graham'], host='localhost')
2024-04-22 04:09:15,693 INFO [archive_tarball] running in process 2096665
2024-04-22 04:09:15,695 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:09:15,696 INFO [archive_tarball] publishing log messages to tcp://*:5564
2024-04-22 04:09:15,697 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:09:15,698 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb20.tar from /ocean/sallen/timbucktwo/oxygen/*feb20/
2024-04-22 04:11:31,205 INFO [download_results] forecast2 2024-04-21 results files from arbutus.cloud-nowcast downloaded
2024-04-22 04:11:31,210 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-22 04:11:31,447 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-22 04:11:34,071 INFO [make_CHS_currents_file] running in process 2096692
2024-04-22 04:11:34,072 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:11:34,073 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2024-04-22 04:11:34,075 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:12:15,293 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-21 for forecast2
2024-04-22 04:12:15,297 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-22 04:12:15,510 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-22 04:12:16,379 INFO [update_forecast_datasets] running in process 2096719
2024-04-22 04:12:16,381 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:12:16,382 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-04-22 04:12:16,383 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:12:16,384 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/21apr24 run
2024-04-22 04:14:07,806 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 390, 72.2% complete
2024-04-22 04:15:25,017 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-feb20.index from /ocean/dlatorne/oxygen-feb20.tar
2024-04-22 04:15:25,127 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-feb20.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-22 04:16:47,714 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/21apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-22 04:16:47,715 INFO [update_forecast_datasets] nemo 2024-04-21 forecast2 rolling forecast datasets updated
2024-04-22 04:16:47,719 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-22 04:16:47,939 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-22 04:16:47,944 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2024-04-21'], host='localhost')
2024-04-22 04:16:47,950 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-22 04:16:48,971 INFO [ping_erddap] running in process 2096800
2024-04-22 04:16:48,972 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:16:48,973 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-22 04:16:48,974 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:16:48,986 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-22 04:16:48,987 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-22 04:16:50,361 INFO [make_surface_current_tiles] running in process 2096803
2024-04-22 04:16:50,364 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:16:50,366 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2024-04-22 04:16:50,367 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:16:50,424 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-21 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/21apr24
2024-04-22 04:16:52,919 INFO [make_plots] running in process 2096802
2024-04-22 04:16:52,921 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:16:52,923 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-22 04:16:52,925 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:16:53,916 INFO [make_plots] preparing render list for 2024-04-21 NEMO forecast2 publish figures
2024-04-22 04:16:54,058 INFO [make_plots] added Website_thumbnail figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,059 INFO [make_plots] added Threshold_website figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,060 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,061 INFO [make_plots] added Boundary Bay figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,062 INFO [make_plots] added Campbell River figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,063 INFO [make_plots] added Cherry Point figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,064 INFO [make_plots] added Friday Harbor figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,065 INFO [make_plots] added Halfmoon Bay figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,066 INFO [make_plots] added Nanaimo figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,067 INFO [make_plots] added Neah Bay figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,068 INFO [make_plots] added New Westminster figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,068 INFO [make_plots] added Patricia Bay figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,069 INFO [make_plots] added Point Atkinson figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,070 INFO [make_plots] added Port Renfrew figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,071 INFO [make_plots] added Sand Heads figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,071 INFO [make_plots] added Sandy Cove figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,072 INFO [make_plots] added Squamish figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,073 INFO [make_plots] added Victoria figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,073 INFO [make_plots] added Woodwards Landing figure to 2024-04-21 NEMO forecast2 publish render list
2024-04-22 04:16:54,074 INFO [make_plots] starting to render nemo forecast2 publish 21apr24 figures
2024-04-22 04:19:07,807 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 462, 85.6% complete
2024-04-22 04:24:07,809 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-22 04:29:07,811 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-22 04:29:07,816 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-22 04:29:08,534 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 04:29:10,075 INFO [download_wwatch3_results] running in process 2098638
2024-04-22 04:29:10,078 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:29:10,079 INFO [download_wwatch3_results] publishing log messages to tcp://*:5568
2024-04-22 04:29:10,080 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:31:24,503 INFO [download_wwatch3_results] forecast2 2024-04-22 results files from arbutus.cloud-nowcast downloaded
2024-04-22 04:31:24,506 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-22 04:31:25,075 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 04:31:27,191 INFO [update_forecast_datasets] running in process 2098977
2024-04-22 04:31:27,192 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:31:27,194 INFO [update_forecast_datasets] publishing log messages to tcp://*:5568
2024-04-22 04:31:27,194 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:31:27,195 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/22apr24 run
2024-04-22 04:31:27,198 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/22apr24 run
2024-04-22 04:32:03,220 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/21apr24
2024-04-22 04:32:06,492 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/22apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-22 04:32:06,493 INFO [update_forecast_datasets] wwatch3 2024-04-22 forecast2 rolling forecast datasets updated
2024-04-22 04:32:06,496 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-22 04:32:06,693 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-22 04:32:07,519 INFO [ping_erddap] running in process 2098991
2024-04-22 04:32:07,520 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:32:07,521 INFO [ping_erddap] publishing log messages to tcp://*:5568
2024-04-22 04:32:07,522 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:32:07,524 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-22 04:32:07,525 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-22 04:32:07,748 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-04-22'], host='localhost')
2024-04-22 04:32:11,075 INFO [make_plots] running in process 2098996
2024-04-22 04:32:11,076 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:32:11,078 INFO [make_plots] publishing log messages to tcp://*:5568
2024-04-22 04:32:11,079 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:32:11,080 INFO [make_plots] preparing render list for 2024-04-22 WaveWatch3 forecast2 publish figures
2024-04-22 04:32:11,081 INFO [make_plots] added Halibut Bank figure to 2024-04-22 WaveWatch3 forecast2 publish render list
2024-04-22 04:32:11,082 INFO [make_plots] added Sentry Shoal figure to 2024-04-22 WaveWatch3 forecast2 publish render list
2024-04-22 04:32:11,083 INFO [make_plots] starting to render wwatch3 forecast2 publish 22apr24 figures
2024-04-22 04:32:11,608 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-22 04:32:18,300 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-22 04:32:24,720 INFO [make_plots] finished rendering wwatch3 forecast2 publish 22apr24 figures
2024-04-22 04:32:24,720 INFO [make_plots] wwatch3 publish plots for 2024-04-22 forecast2 completed
2024-04-22 04:32:24,724 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-22 04:35:49,873 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-22 04:35:49,875 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-22 04:35:52,754 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/21apr24
2024-04-22 04:35:52,764 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-21 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/21apr24
2024-04-22 04:35:52,765 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-21 forecast2 completed
2024-04-22 04:35:53,064 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-22 04:36:26,324 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:38:18,429 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:38:18,845 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:38:56,149 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:39:38,117 INFO [archive_tarball] hindcast *feb20 results files archived to robot.graham
2024-04-22 04:39:38,120 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-22 04:40:08,698 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:40:45,538 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:41:21,756 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:41:58,281 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:42:34,437 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:43:10,953 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:43:48,045 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:43:48,451 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:44:24,012 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:45:00,053 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-17 00:00:00Z to 2024-04-24 05:55:00Z
2024-04-22 04:45:36,395 INFO [make_plots] finished rendering nemo forecast2 publish 21apr24 figures
2024-04-22 04:45:36,397 INFO [make_plots] nemo publish plots for 2024-04-21 forecast2 completed
2024-04-22 04:45:36,406 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-22 04:45:36,765 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2024-04-21'], host='localhost')
2024-04-22 04:45:39,793 INFO [make_feeds] running in process 2099191
2024-04-22 04:45:39,795 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:45:39,796 INFO [make_feeds] publishing log messages to tcp://*:5562
2024-04-22 04:45:39,797 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:46:07,275 INFO [make_feeds] ATOM feeds for 2024-04-21 forecast2 run completed
2024-04-22 04:46:07,278 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-22 04:46:07,616 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2024-04-22 04:46:08,367 INFO [clear_checklist] running in process 2099215
2024-04-22 04:46:08,369 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:46:08,370 INFO [clear_checklist] publishing log messages to tcp://*:5562
2024-04-22 04:46:08,372 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:46:08,373 INFO [clear_checklist] requesting that manager clear system state checklist
2024-04-22 04:46:08,375 INFO [manager] writing checklist to log file
2024-04-22 04:46:08,424 INFO [manager] checklist cleared
2024-04-22 04:46:08,427 INFO [clear_checklist] nowcast system checklist cleared
2024-04-22 04:46:08,430 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2024-04-22 04:46:09,345 INFO [rotate_logs] running in process 2099220
2024-04-22 04:46:09,347 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-22 04:46:09,348 INFO [rotate_logs] publishing log messages to tcp://*:5562
2024-04-22 04:46:09,349 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2024-04-22 04:46:09,349 INFO [rotate_logs] rotating log files