2024-04-19 04:49:26,877 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2024-04-19 04:49:26,885 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2024-04-19 04:49:26,890 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2024-04-19 04:49:26,892 INFO [rotate_logs] log files rotated
2024-04-19 04:49:26,896 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2024-04-19 05:57:39,024 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 05:57:39,332 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-07-26'], host='localhost')
2024-04-19 05:57:39,339 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 05:57:39,343 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 05:57:40,721 INFO [download_results] running in process 2041029
2024-04-19 05:57:40,723 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 05:57:40,723 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 05:57:40,725 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 05:57:40,727 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26jul19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 05:57:44,876 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 05:59:24,549 INFO [download_results] hindcast 2019-07-26 results files from optimum-hindcast downloaded
2024-04-19 05:59:24,603 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 05:59:24,609 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-07-26'], host='localhost')
2024-04-19 05:59:25,758 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 05:59:25,771 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-jul', 'robot.graham'], host='localhost')
2024-04-19 05:59:26,785 INFO [archive_tarball] running in process 2041077
2024-04-19 05:59:26,787 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 05:59:26,788 INFO [archive_tarball] publishing log messages to tcp://*:5562
2024-04-19 05:59:26,789 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-19 05:59:26,790 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jul19.tar from /ocean/sallen/timbucktwo/oxygen/*jul19/
2024-04-19 06:05:48,313 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-jul19.index from /ocean/dlatorne/oxygen-jul19.tar
2024-04-19 06:05:48,423 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-jul19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-19 06:31:18,263 INFO [archive_tarball] hindcast *jul19 results files archived to robot.graham
2024-04-19 06:31:18,266 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-19 07:22:49,321 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 07:22:49,557 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-01'], host='localhost')
2024-04-19 07:22:49,563 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 07:22:49,568 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 07:22:51,040 INFO [download_results] running in process 2045833
2024-04-19 07:22:51,042 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 07:22:51,043 INFO [download_results] publishing log messages to tcp://*:5565
2024-04-19 07:22:51,044 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 07:22:51,045 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01aug19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 07:22:55,260 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 07:23:54,603 INFO [download_results] hindcast 2019-08-01 results files from optimum-hindcast downloaded
2024-04-19 07:23:54,615 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 07:23:54,623 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-01'], host='localhost')
2024-04-19 07:23:55,972 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 07:58:58,609 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-19 07:58:58,622 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-19'], host='arbutus.cloud-nowcast')
2024-04-19 07:59:00,638 INFO [run_ww3] running in process 13314
2024-04-19 07:59:00,639 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 07:59:00,640 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-19 07:59:00,641 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-19 07:59:00,646 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2024-04-19T075900.634840-0700
2024-04-19 07:59:00,648 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2024-04-19T075900.634840-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-19 07:59:00,663 INFO [run_ww3] forecast2 WaveWatch3 run for 2024-04-19 on arbutus.cloud-nowcast started
2024-04-19 07:59:00,667 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-19 07:59:00,680 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2024-04-19 07:59:02,606 INFO [watch_ww3] running in process 13404
2024-04-19 07:59:02,631 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 07:59:02,635 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-04-19 07:59:02,636 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-19 07:59:02,640 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-19 08:04:02,640 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2024-04-19 08:09:02,639 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 198, 36.7% complete
2024-04-19 08:14:02,639 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 300, 55.6% complete
2024-04-19 08:19:02,639 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 399, 73.9% complete
2024-04-19 08:24:02,642 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 498, 92.2% complete
2024-04-19 08:29:02,641 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-19 08:29:02,690 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-19 08:29:02,913 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 08:29:04,129 INFO [download_wwatch3_results] running in process 2046147
2024-04-19 08:29:04,130 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 08:29:04,132 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2024-04-19 08:29:04,133 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 08:31:23,122 INFO [download_wwatch3_results] forecast2 2024-04-19 results files from arbutus.cloud-nowcast downloaded
2024-04-19 08:31:23,125 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-19 08:31:23,135 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 08:31:24,265 INFO [update_forecast_datasets] running in process 2046312
2024-04-19 08:31:24,267 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 08:31:24,268 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2024-04-19 08:31:24,269 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-19 08:31:24,270 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/19apr24 run
2024-04-19 08:31:24,271 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/19apr24 run
2024-04-19 08:31:49,641 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/19apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-19 08:31:49,658 INFO [update_forecast_datasets] wwatch3 2024-04-19 forecast2 rolling forecast datasets updated
2024-04-19 08:31:49,694 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-19 08:31:49,707 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-19 08:31:50,604 INFO [ping_erddap] running in process 2046324
2024-04-19 08:31:50,605 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 08:31:50,607 INFO [ping_erddap] publishing log messages to tcp://*:5562
2024-04-19 08:31:50,608 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-19 08:31:50,609 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-19 08:31:50,611 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-19 08:31:50,623 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 08:31:58,434 INFO [make_plots] running in process 2046329
2024-04-19 08:31:58,435 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 08:31:58,436 INFO [make_plots] publishing log messages to tcp://*:5562
2024-04-19 08:31:58,437 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-19 08:31:58,439 INFO [make_plots] preparing render list for 2024-04-19 WaveWatch3 forecast2 publish figures
2024-04-19 08:31:58,440 INFO [make_plots] added Halibut Bank figure to 2024-04-19 WaveWatch3 forecast2 publish render list
2024-04-19 08:31:58,440 INFO [make_plots] added Sentry Shoal figure to 2024-04-19 WaveWatch3 forecast2 publish render list
2024-04-19 08:31:58,441 INFO [make_plots] starting to render wwatch3 forecast2 publish 19apr24 figures
2024-04-19 08:31:58,881 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-19 08:32:08,620 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-19 08:32:14,360 INFO [make_plots] finished rendering wwatch3 forecast2 publish 19apr24 figures
2024-04-19 08:32:14,373 INFO [make_plots] wwatch3 publish plots for 2024-04-19 forecast2 completed
2024-04-19 08:32:14,375 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-19 08:47:59,333 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 08:47:59,579 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-06'], host='localhost')
2024-04-19 08:47:59,585 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 08:47:59,589 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 08:48:01,031 INFO [download_results] running in process 2046401
2024-04-19 08:48:01,032 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 08:48:01,033 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 08:48:01,034 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 08:48:01,035 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06aug19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 08:48:05,310 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 08:49:01,091 INFO [download_results] hindcast 2019-08-06 results files from optimum-hindcast downloaded
2024-04-19 08:49:01,107 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 08:49:01,116 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-06'], host='localhost')
2024-04-19 08:49:02,380 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 09:11:08,887 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20240419/12/
2024-04-19 09:11:08,888 INFO [collect_weather] 2024-04-19 2.5km weather forecast 12 collection complete
2024-04-19 09:11:08,890 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-19 09:11:09,111 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2024-04-18'], host='localhost')
2024-04-19 09:11:09,116 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2024-04-18'], host='localhost')
2024-04-19 09:11:09,121 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2024-04-18'], host='localhost')
2024-04-19 09:11:09,127 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2024-04-18'], host='localhost')
2024-04-19 09:11:09,131 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2024-04-19 09:11:09,135 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2024-04-19 09:11:09,142 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2024-04-19 09:11:09,146 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2024-04-19 09:11:09,149 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2024-04-19 09:11:10,630 INFO [collect_NeahBay_ssh] running in process 2046738
2024-04-19 09:11:10,631 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:10,634 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2024-04-19 09:11:10,635 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:10,637 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2024-04-19
2024-04-19 09:11:10,638 INFO [collect_weather] running in process 2046740
2024-04-19 09:11:10,639 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:10,640 INFO [collect_weather] publishing log messages to tcp://*:5563
2024-04-19 09:11:10,641 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:10,653 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2024-04-19 09:11:10,984 INFO [download_live_ocean] running in process 2046739
2024-04-19 09:11:11,179 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:11,202 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2024-04-19 09:11:11,225 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:11,235 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2024-04-19
2024-04-19 09:11:12,582 INFO [collect_NeahBay_ssh] 2024-04-19T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2024-04-19 09:11:12,586 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2024-04-19 09:11:12,599 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 09:11:12,651 INFO [collect_river_data] running in process 2046734
2024-04-19 09:11:12,652 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:12,653 INFO [collect_river_data] publishing log messages to tcp://*:5566
2024-04-19 09:11:12,654 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:12,654 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2024-04-18
2024-04-19 09:11:12,673 INFO [collect_river_data] running in process 2046732
2024-04-19 09:11:12,674 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:12,675 INFO [collect_river_data] publishing log messages to tcp://*:5567
2024-04-19 09:11:12,676 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:12,677 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2024-04-18
2024-04-19 09:11:12,688 INFO [collect_river_data] running in process 2046735
2024-04-19 09:11:12,689 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:12,690 INFO [collect_river_data] publishing log messages to tcp://*:5569
2024-04-19 09:11:12,691 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:12,691 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2024-04-18
2024-04-19 09:11:12,699 INFO [collect_river_data] running in process 2046736
2024-04-19 09:11:12,700 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:12,701 INFO [collect_river_data] publishing log messages to tcp://*:5570
2024-04-19 09:11:12,702 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:12,703 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2024-04-18
2024-04-19 09:11:12,746 INFO [make_turbidity_file] running in process 2046737
2024-04-19 09:11:12,747 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:12,748 INFO [make_turbidity_file] publishing log messages to tcp://*:5571
2024-04-19 09:11:12,749 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:12,750 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2024-04-19
2024-04-19 09:11:12,807 INFO [download_live_ocean] downloaded 10102357 bytes from https://liveocean.apl.uw.edu/output/f2024.04.19/ubc.nc to /results/forcing/LiveOcean/downloaded/20240419/low_passed_UBC.nc
2024-04-19 09:11:12,808 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2024-04-19 09:11:12,887 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2024-04-18&endDT=2024-04-18 "HTTP/1.1 200 OK"
2024-04-19 09:11:12,888 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2024-04-18&endDT=2024-04-18 "HTTP/1.1 200 OK"
2024-04-19 09:11:12,902 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2024-04-18 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2024-04-19 09:11:12,903 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2024-04-18 completed
2024-04-19 09:11:12,904 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2024-04-18 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2024-04-19 09:11:12,906 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2024-04-18 completed
2024-04-19 09:11:12,909 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-19 09:11:12,916 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-19 09:11:13,026 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2024-04-18&endDT=2024-04-18 "HTTP/1.1 200 OK"
2024-04-19 09:11:13,103 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2024-04-18&endDT=2024-04-18 "HTTP/1.1 200 OK"
2024-04-19 09:11:13,260 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2024-04-18 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2024-04-19 09:11:13,260 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2024-04-18 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2024-04-19 09:11:13,261 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2024-04-18 completed
2024-04-19 09:11:13,262 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2024-04-18 completed
2024-04-19 09:11:13,264 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-19 09:11:13,280 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2024-04-19 09:11:13,512 INFO [crop_gribs] running in process 2046741
2024-04-19 09:11:13,515 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:13,516 INFO [crop_gribs] publishing log messages to tcp://*:5562
2024-04-19 09:11:13,517 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:13,518 INFO [crop_gribs] cropping 2024-04-19 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2024-04-19 09:11:13,527 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240419/18/
2024-04-19 09:11:13,890 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20240419/low_passed_UBC.nc
2024-04-19 09:11:13,891 INFO [download_live_ocean] 2024-04-19 Live Ocean file for Salish Sea western boundary downloaded
2024-04-19 09:11:13,893 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2024-04-19 09:11:14,116 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2024-04-19'], host='localhost')
2024-04-19 09:11:16,134 INFO [make_turbidity_file] 2024-04-19 Fraser River turbidity file creation complete
2024-04-19 09:11:16,170 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2024-04-19 09:11:16,728 INFO [make_ssh_files] running in process 2046881
2024-04-19 09:11:16,743 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:16,744 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2024-04-19 09:11:16,745 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:16,746 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2024-04-19 NOAA Neah Bay 06Z observation and forecast values
2024-04-19 09:11:16,781 INFO [make_live_ocean_files] running in process 2046905
2024-04-19 09:11:16,782 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:16,784 INFO [make_live_ocean_files] publishing log messages to tcp://*:5566
2024-04-19 09:11:16,785 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:16,786 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2024-04-19 Live Ocean run
2024-04-19 09:11:21,970 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d15.nc
2024-04-19 09:11:22,180 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d16.nc
2024-04-19 09:11:22,378 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d17.nc
2024-04-19 09:11:22,548 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2024m04d18.nc
2024-04-19 09:11:22,746 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d19.nc
2024-04-19 09:11:22,928 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d20.nc
2024-04-19 09:11:23,121 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d21.nc
2024-04-19 09:11:23,278 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2024m04d22.nc
2024-04-19 09:11:23,432 INFO [make_ssh_files] sea surface height boundary file for 2024-04-19 nowcast run created
2024-04-19 09:11:23,435 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2024-04-19 09:11:23,444 INFO [manager] launching NextWorker(module='nowcast.workers.make_v202111_runoff_file', args=[], host='localhost')
2024-04-19 09:11:23,448 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2024-04-19 09:11:24,667 INFO [make_runoff_file] running in process 2046951
2024-04-19 09:11:24,668 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:24,669 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2024-04-19 09:11:24,670 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:24,762 INFO [make_runoff_file] calculating runoff forcing for b201702 bathymetry
2024-04-19 09:11:24,879 INFO [make_runoff_file] 2024-04-19 runoff file creation from Fraser at Hope and climatology elsewhere complete
2024-04-19 09:11:24,881 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2024-04-19 09:11:25,639 INFO [make_v202111_runoff_file] running in process 2046949
2024-04-19 09:11:25,640 INFO [make_v202111_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:11:25,642 INFO [make_v202111_runoff_file] publishing log messages to tcp://*:5564
2024-04-19 09:11:25,643 INFO [make_v202111_runoff_file] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:11:25,644 INFO [make_v202111_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2024-04-18
2024-04-19 09:11:28,720 INFO [make_v202111_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2024-04-18: /results/forcing/rivers/R202108Dailies_y2024m04d18.nc
2024-04-19 09:11:28,721 INFO [make_v202111_runoff_file] 2024-04-18 runoff file creation completed
2024-04-19 09:11:28,723 INFO [manager] checklist updated with [v202111 rivers forcing] items from make_v202111_runoff_file worker
2024-04-19 09:12:10,505 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /
2024-04-19 09:12:10,506 INFO [make_live_ocean_files] 2024-04-19 Live Ocean western boundary conditions files created
2024-04-19 09:12:10,508 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2024-04-19 09:14:25,947 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240419/12/
2024-04-19 09:14:25,966 INFO [crop_gribs] 2024-04-19 12 GRIBs cropping complete
2024-04-19 09:14:25,979 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-19 09:14:26,241 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2024-04-19 09:14:28,531 INFO [grib_to_netcdf] running in process 2046995
2024-04-19 09:14:28,532 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:14:28,534 INFO [grib_to_netcdf] publishing log messages to tcp://*:5564
2024-04-19 09:14:28,535 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:14:28,536 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2024-04-19 nowcast and forecast runs
2024-04-19 09:14:43,271 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2024m04d19.nc
2024-04-19 09:14:54,525 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d20.nc
2024-04-19 09:15:00,770 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2024m04d21.nc
2024-04-19 09:15:00,786 INFO [grib_to_netcdf] 2024-04-19 NEMO-atmos forcing file for nowcast+ created
2024-04-19 09:15:00,789 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2024-04-19 09:15:00,803 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2024-04-19 09:15:00,806 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2024-04-19 09:15:00,812 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2024-04-19 09:15:00,816 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2024-04-19 09:15:00,819 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2024-04-19 09:15:01,915 INFO [ping_erddap] running in process 2047038
2024-04-19 09:15:01,918 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:15:01,919 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-04-19 09:15:01,920 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:15:01,922 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2024-04-19 09:15:01,924 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-19 09:15:01,964 INFO [upload_forcing] running in process 2047035
2024-04-19 09:15:01,966 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:15:01,967 INFO [upload_forcing] publishing log messages to tcp://*:5566
2024-04-19 09:15:01,968 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:15:01,978 INFO [upload_forcing] running in process 2047036
2024-04-19 09:15:01,979 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:15:01,981 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-19 09:15:01,982 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:15:01,990 INFO [upload_forcing] running in process 2047033
2024-04-19 09:15:01,991 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:15:01,992 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-19 09:15:01,993 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:15:02,204 INFO [upload_forcing] running in process 2047037
2024-04-19 09:15:02,206 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:15:02,207 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-04-19 09:15:02,208 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:15:04,524 INFO [upload_forcing] nowcast+ 2024-04-19 forcing files upload to optimum-hindcast completed
2024-04-19 09:15:04,531 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-19 09:15:04,823 INFO [upload_forcing] nowcast+ 2024-04-19 forcing files upload to orcinus-nowcast-agrif completed
2024-04-19 09:15:04,826 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-19 09:15:11,414 INFO [upload_forcing] nowcast+ 2024-04-19 forcing files upload to robot.graham completed
2024-04-19 09:15:11,419 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-19 09:15:14,448 INFO [upload_forcing] nowcast+ 2024-04-19 forcing files upload to arbutus.cloud-nowcast completed
2024-04-19 09:15:14,476 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-19 09:15:14,512 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 09:15:15,618 INFO [make_forcing_links] running in process 2047072
2024-04-19 09:15:15,619 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:15:15,646 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-04-19 09:15:15,664 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:15:16,709 INFO [make_forcing_links] nowcast+ 2024-04-19 forcing file links on arbutus.cloud-nowcast created
2024-04-19 09:15:16,713 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-19 09:15:16,731 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-19'], host='arbutus.cloud-nowcast')
2024-04-19 09:15:18,751 INFO [run_NEMO] running in process 17767
2024-04-19 09:15:18,768 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:15:18,769 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-19 09:15:18,770 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:15:19,629 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/19apr24nowcast_2024-04-19T091518.802825-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-19 09:15:19,649 INFO [run_NEMO] nowcast NEMO run for 2024-04-19 on arbutus.cloud-nowcast started
2024-04-19 09:15:19,653 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-19 09:15:19,668 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-19 09:15:21,623 INFO [watch_NEMO] running in process 18021
2024-04-19 09:15:21,624 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:15:21,625 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-19 09:15:21,626 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:15:21,693 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-19 09:20:21,694 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17597950 = 2024-04-19 04:46:00 UTC, 19.9% complete
2024-04-19 09:25:21,695 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17598442 = 2024-04-19 10:14:00 UTC, 42.7% complete
2024-04-19 09:30:21,696 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17598917 = 2024-04-19 15:30:40 UTC, 64.7% complete
2024-04-19 09:35:21,697 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 17599391 = 2024-04-19 20:46:40 UTC, 86.6% complete
2024-04-19 09:40:21,701 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2024-04-19 09:40:21,704 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-19 09:40:21,917 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 09:40:21,924 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 09:40:23,209 INFO [download_results] running in process 2047199
2024-04-19 09:40:23,210 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:40:23,212 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 09:40:23,213 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:40:23,214 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/19apr24 to /results/SalishSea/nowcast-blue.202111
2024-04-19 09:40:23,216 INFO [make_forcing_links] running in process 2047198
2024-04-19 09:40:23,217 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:40:23,218 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2024-04-19 09:40:23,219 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:40:24,159 INFO [make_forcing_links] ssh 2024-04-19 forcing file links on arbutus.cloud-nowcast created
2024-04-19 09:40:24,163 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-19 09:40:24,180 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-19'], host='arbutus.cloud-nowcast')
2024-04-19 09:40:26,442 INFO [run_NEMO] running in process 19247
2024-04-19 09:40:26,464 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:40:26,466 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-19 09:40:26,467 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:40:27,392 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/19apr24forecast_2024-04-19T094026.514018-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-19 09:40:27,410 INFO [run_NEMO] forecast NEMO run for 2024-04-19 on arbutus.cloud-nowcast started
2024-04-19 09:40:27,415 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-19 09:40:27,436 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-19 09:40:29,569 INFO [watch_NEMO] running in process 19505
2024-04-19 09:40:29,571 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 09:40:29,572 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-19 09:40:29,573 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-19 09:40:29,744 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-19 09:45:29,737 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17600122 = 2024-04-20 04:54:00 UTC, 13.6% complete
2024-04-19 09:50:29,738 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17600609 = 2024-04-20 10:18:40 UTC, 28.7% complete
2024-04-19 09:55:29,765 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17601105 = 2024-04-20 15:49:20 UTC, 44.0% complete
2024-04-19 10:00:29,730 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17601584 = 2024-04-20 21:08:40 UTC, 58.8% complete
2024-04-19 10:00:29,934 INFO [download_results] nowcast 2024-04-19 results files from arbutus.cloud-nowcast downloaded
2024-04-19 10:00:29,939 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 10:00:29,955 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:00:29,962 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2024-04-18'], host='localhost')
2024-04-19 10:00:29,966 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:00:32,955 INFO [make_CHS_currents_file] running in process 2047284
2024-04-19 10:00:32,956 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:00:32,957 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2024-04-19 10:00:32,958 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:00:36,725 INFO [make_plots] running in process 2047281
2024-04-19 10:00:36,726 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:00:36,727 INFO [make_plots] publishing log messages to tcp://*:5566
2024-04-19 10:00:36,728 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:00:36,734 INFO [make_plots] running in process 2047283
2024-04-19 10:00:36,736 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:00:36,737 INFO [make_plots] publishing log messages to tcp://*:5567
2024-04-19 10:00:36,738 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:00:37,461 INFO [make_plots] preparing render list for 2024-04-18 NEMO nowcast-blue comparison figures
2024-04-19 10:00:37,483 INFO [make_plots] preparing render list for 2024-04-19 NEMO nowcast-blue research figures
2024-04-19 10:00:38,695 INFO [make_plots] added salinity_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-blue research render list
2024-04-19 10:00:39,781 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-blue research render list
2024-04-19 10:00:39,782 INFO [make_plots] added Currents_sections_and_surface figure to 2024-04-19 NEMO nowcast-blue research render list
2024-04-19 10:00:39,783 INFO [make_plots] added Currents_at_VENUS_Central figure to 2024-04-19 NEMO nowcast-blue research render list
2024-04-19 10:00:39,783 INFO [make_plots] added Currents_at_VENUS_East figure to 2024-04-19 NEMO nowcast-blue research render list
2024-04-19 10:00:39,784 INFO [make_plots] starting to render nemo nowcast research 19apr24 figures
2024-04-19 10:00:58,790 INFO [make_plots] added SH_wind figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,791 INFO [make_plots] added Compare_VENUS_East figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,792 INFO [make_plots] added Compare_VENUS_Central figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,793 INFO [make_plots] added Central_ADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,794 INFO [make_plots] added Central_depavADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,795 INFO [make_plots] added Central_timeavADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,796 INFO [make_plots] added East_ADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,797 INFO [make_plots] added East_depavADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,798 INFO [make_plots] added East_timeavADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,799 INFO [make_plots] added ddl_ADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,799 INFO [make_plots] added ddl_depavADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,800 INFO [make_plots] added ddl_timeavADCP figure to 2024-04-18 NEMO nowcast-blue comparison render list
2024-04-19 10:00:58,801 INFO [make_plots] starting to render nemo nowcast comparison 18apr24 figures
2024-04-19 10:01:10,936 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-19 for nowcast
2024-04-19 10:01:10,939 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-19 10:02:01,006 INFO [make_plots] finished rendering nemo nowcast research 19apr24 figures
2024-04-19 10:02:01,008 INFO [make_plots] nemo research plots for 2024-04-19 nowcast completed
2024-04-19 10:02:01,028 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-19 10:03:04,736 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-19 10:03:05,233 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-19 10:03:05,725 WARNING [make_plots] VENUS Central ADCP comparison figure failed: No observations available
2024-04-19 10:03:06,221 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-19 10:03:06,717 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-19 10:03:07,213 WARNING [make_plots] VENUS East ADCP comparison figure failed: No observations available
2024-04-19 10:03:07,719 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-19 10:03:08,211 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-19 10:03:08,705 WARNING [make_plots] VENUS ddl ADCP comparison figure failed: No observations available
2024-04-19 10:03:08,707 INFO [make_plots] finished rendering nemo nowcast comparison 18apr24 figures
2024-04-19 10:03:08,812 INFO [make_plots] nemo comparison plots for 2024-04-18 nowcast completed
2024-04-19 10:03:08,815 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-19 10:05:29,731 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17602067 = 2024-04-21 02:30:40 UTC, 73.7% complete
2024-04-19 10:10:29,732 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17602574 = 2024-04-21 08:08:40 UTC, 89.3% complete
2024-04-19 10:15:29,733 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 17602920 = 2024-04-21 11:59:20 UTC, 100.0% complete
2024-04-19 10:18:10,161 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 10:18:10,374 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-11'], host='localhost')
2024-04-19 10:18:10,380 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 10:18:10,384 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 10:18:11,720 INFO [download_results] running in process 2048096
2024-04-19 10:18:11,721 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:18:11,722 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 10:18:11,723 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:18:11,725 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11aug19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 10:18:15,943 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 10:19:14,815 INFO [download_results] hindcast 2019-08-11 results files from optimum-hindcast downloaded
2024-04-19 10:19:14,830 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 10:19:14,870 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-11'], host='localhost')
2024-04-19 10:19:16,166 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 10:20:29,737 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2024-04-19 10:20:29,741 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-19 10:20:29,958 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-19'], host='arbutus.cloud-nowcast')
2024-04-19 10:20:29,966 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-19'], host='arbutus.cloud-nowcast')
2024-04-19 10:20:29,970 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:20:29,975 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:20:29,981 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:20:29,985 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:20:29,990 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:20:31,004 INFO [upload_forcing] running in process 2048147
2024-04-19 10:20:31,005 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:31,006 INFO [upload_forcing] publishing log messages to tcp://*:5564
2024-04-19 10:20:31,007 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:31,009 INFO [upload_forcing] running in process 2048149
2024-04-19 10:20:31,010 INFO [upload_forcing] running in process 2048148
2024-04-19 10:20:31,011 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:31,012 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:31,013 INFO [upload_forcing] publishing log messages to tcp://*:5565
2024-04-19 10:20:31,014 INFO [upload_forcing] publishing log messages to tcp://*:5567
2024-04-19 10:20:31,015 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:31,016 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:31,016 INFO [upload_forcing] running in process 2048146
2024-04-19 10:20:31,017 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:31,018 INFO [upload_forcing] publishing log messages to tcp://*:5568
2024-04-19 10:20:31,019 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:31,020 INFO [download_results] running in process 2048150
2024-04-19 10:20:31,020 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:31,021 INFO [download_results] publishing log messages to tcp://*:5569
2024-04-19 10:20:31,022 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:31,023 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/19apr24 to /results/SalishSea/forecast.202111
2024-04-19 10:20:31,776 INFO [upload_forcing] turbidity 2024-04-19 forcing files upload to optimum-hindcast completed
2024-04-19 10:20:31,781 INFO [upload_forcing] turbidity 2024-04-19 forcing files upload to orcinus-nowcast-agrif completed
2024-04-19 10:20:31,782 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-19 10:20:31,804 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-19 10:20:31,822 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:20:32,059 INFO [upload_forcing] turbidity 2024-04-19 forcing files upload to arbutus.cloud-nowcast completed
2024-04-19 10:20:32,063 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-19 10:20:32,085 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:20:32,783 INFO [make_forcing_links] running in process 2048181
2024-04-19 10:20:32,784 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:32,786 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2024-04-19 10:20:32,787 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:33,067 INFO [make_forcing_links] running in process 2048183
2024-04-19 10:20:33,068 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:33,070 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2024-04-19 10:20:33,071 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:33,804 INFO [make_forcing_links] nowcast-agrif 2024-04-19 forcing file links on orcinus-nowcast-agrif created
2024-04-19 10:20:33,808 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-19 10:20:33,829 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:20:34,802 INFO [run_NEMO_agrif] running in process 2048197
2024-04-19 10:20:34,803 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:34,805 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2024-04-19 10:20:34,806 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:35,090 INFO [upload_forcing] turbidity 2024-04-19 forcing files upload to robot.graham completed
2024-04-19 10:20:35,095 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2024-04-19 10:20:36,810 INFO [make_forcing_links] nowcast-green 2024-04-19 forcing file links on arbutus.cloud-nowcast created
2024-04-19 10:20:36,814 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2024-04-19 10:20:36,838 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-19'], host='arbutus.cloud-nowcast')
2024-04-19 10:20:37,471 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2024-04-19 10:20:39,432 INFO [run_NEMO] running in process 21850
2024-04-19 10:20:39,433 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:39,435 INFO [run_NEMO] publishing log messages to tcp://*:5556
2024-04-19 10:20:39,436 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:40,340 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/19apr24nowcast-green_2024-04-19T102039.458343-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2024-04-19 10:20:40,358 INFO [run_NEMO] nowcast-green NEMO run for 2024-04-19 on arbutus.cloud-nowcast started
2024-04-19 10:20:40,363 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2024-04-19 10:20:40,385 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2024-04-19 10:20:42,679 INFO [watch_NEMO] running in process 22105
2024-04-19 10:20:42,680 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:42,681 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2024-04-19 10:20:42,682 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:42,792 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2024-04-19 10:20:57,174 INFO [run_NEMO_agrif] job id for 19apr24nowcast-agrif: 11091794.orca2.ibb
2024-04-19 10:20:57,175 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2024-04-19 queued on orcinus-nowcast-agrif
2024-04-19 10:20:57,181 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2024-04-19 10:20:57,204 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11091794.orca2.ibb'], host='localhost')
2024-04-19 10:20:58,449 INFO [watch_NEMO_agrif] running in process 2048216
2024-04-19 10:20:58,450 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:20:58,451 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2024-04-19 10:20:58,452 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:20:59,278 INFO [watch_NEMO_agrif] watching 19apr24nowcast-agrif job 11091794 on orcinus-nowcast-agrif
2024-04-19 10:20:59,516 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif job 11091794 is queued on orcinus-nowcast-agrif
2024-04-19 10:21:16,300 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2024-04-19 10:21:16,326 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-19'], host='arbutus.cloud-nowcast')
2024-04-19 10:21:18,493 INFO [run_ww3] running in process 22207
2024-04-19 10:21:18,506 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:21:18,507 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-19 10:21:18,508 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:21:18,512 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-19T102118.478778-0700
2024-04-19 10:21:18,513 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2024-04-19T102118.478778-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-19 10:21:18,519 INFO [run_ww3] nowcast WaveWatch3 run for 2024-04-19 on arbutus.cloud-nowcast started
2024-04-19 10:21:18,551 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-19 10:21:18,575 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2024-04-19 10:21:20,659 INFO [watch_ww3] running in process 22295
2024-04-19 10:21:20,661 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:21:20,662 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-04-19 10:21:20,663 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:21:20,751 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-19 10:22:00,585 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2024-04-19 10:25:42,793 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17597683 = 2024-04-19 01:48:00 UTC, 7.5% complete
2024-04-19 10:26:20,743 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2024-04-19 10:27:01,031 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7618506 = 2024-04-19 02:03:20 UTC, 8.6% complete
2024-04-19 10:30:42,799 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17597873 = 2024-04-19 03:54:40 UTC, 16.3% complete
2024-04-19 10:31:20,747 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 198, 45.8% complete
2024-04-19 10:32:01,516 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7618701 = 2024-04-19 04:13:20 UTC, 17.6% complete
2024-04-19 10:35:42,882 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17598058 = 2024-04-19 05:58:00 UTC, 24.9% complete
2024-04-19 10:36:20,745 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 300, 69.4% complete
2024-04-19 10:37:01,969 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7618898 = 2024-04-19 06:24:40 UTC, 26.7% complete
2024-04-19 10:40:42,793 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17598248 = 2024-04-19 08:04:40 UTC, 33.7% complete
2024-04-19 10:41:20,745 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 408, 94.4% complete
2024-04-19 10:42:02,451 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7619096 = 2024-04-19 08:36:40 UTC, 35.9% complete
2024-04-19 10:45:42,791 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17598429 = 2024-04-19 10:05:20 UTC, 42.1% complete
2024-04-19 10:46:20,749 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-19 10:46:20,753 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-19 10:46:20,974 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:46:20,980 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-19'], host='arbutus.cloud-nowcast')
2024-04-19 10:46:22,175 INFO [download_wwatch3_results] running in process 2048265
2024-04-19 10:46:22,178 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:46:22,179 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-04-19 10:46:22,180 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:46:23,165 INFO [run_ww3] running in process 23442
2024-04-19 10:46:23,166 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:46:23,167 INFO [run_ww3] publishing log messages to tcp://*:5572
2024-04-19 10:46:23,169 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:46:23,173 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-19T104623.153502-0700
2024-04-19 10:46:23,184 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2024-04-19T104623.153502-0700/SoGWW3.sh on arbutus.cloud-nowcast
2024-04-19 10:46:23,192 INFO [run_ww3] forecast WaveWatch3 run for 2024-04-19 on arbutus.cloud-nowcast started
2024-04-19 10:46:23,225 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2024-04-19 10:46:23,251 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2024-04-19 10:46:25,400 INFO [watch_ww3] running in process 23598
2024-04-19 10:46:25,401 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:46:25,402 INFO [watch_ww3] publishing log messages to tcp://*:5573
2024-04-19 10:46:25,404 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:46:25,586 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2024-04-19 10:47:02,931 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7619294 = 2024-04-19 10:48:40 UTC, 45.1% complete
2024-04-19 10:50:41,278 INFO [download_wwatch3_results] nowcast 2024-04-19 results files from arbutus.cloud-nowcast downloaded
2024-04-19 10:50:41,281 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-19 10:50:42,793 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17598612 = 2024-04-19 12:07:20 UTC, 50.5% complete
2024-04-19 10:51:25,566 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2024-04-19 10:52:00,044 INFO [download_results] forecast 2024-04-19 results files from arbutus.cloud-nowcast downloaded
2024-04-19 10:52:00,048 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 10:52:00,071 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:52:02,775 INFO [make_CHS_currents_file] running in process 2048286
2024-04-19 10:52:02,797 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:52:02,815 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5565
2024-04-19 10:52:02,821 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:52:03,382 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7619489 = 2024-04-19 12:58:40 UTC, 54.1% complete
2024-04-19 10:52:50,201 INFO [make_CHS_currents_file] Made CHS currents file for 2024-04-19 for forecast
2024-04-19 10:52:50,203 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2024-04-19 10:52:50,223 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:52:51,259 INFO [update_forecast_datasets] running in process 2048311
2024-04-19 10:52:51,261 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:52:51,262 INFO [update_forecast_datasets] publishing log messages to tcp://*:5565
2024-04-19 10:52:51,262 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:52:51,263 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/19apr24 run
2024-04-19 10:52:51,493 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/19apr24 run: /results/SalishSea/rolling-forecasts/nemo
2024-04-19 10:52:51,494 INFO [update_forecast_datasets] nemo 2024-04-19 forecast rolling forecast datasets updated
2024-04-19 10:52:51,496 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-19 10:52:51,514 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2024-04-19 10:52:51,518 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:52:51,522 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 10:52:52,564 INFO [ping_erddap] running in process 2048316
2024-04-19 10:52:52,566 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:52:52,567 INFO [ping_erddap] publishing log messages to tcp://*:5565
2024-04-19 10:52:52,568 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:52:52,577 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2024-04-19 10:52:52,578 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-19 10:52:54,387 INFO [make_surface_current_tiles] running in process 2048319
2024-04-19 10:52:54,388 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:52:54,389 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5565
2024-04-19 10:52:54,390 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:52:54,392 INFO [make_surface_current_tiles] starting rendering of tiles for 2024-04-19 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/19apr24
2024-04-19 10:52:58,633 INFO [make_plots] running in process 2048318
2024-04-19 10:52:58,658 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 10:52:58,676 INFO [make_plots] publishing log messages to tcp://*:5567
2024-04-19 10:52:58,680 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-19 10:52:59,516 INFO [make_plots] preparing render list for 2024-04-19 NEMO forecast publish figures
2024-04-19 10:52:59,694 INFO [make_plots] added Website_thumbnail figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,695 INFO [make_plots] added Threshold_website figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,696 INFO [make_plots] added PA_tidal_predictions figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,697 INFO [make_plots] added Boundary Bay figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,698 INFO [make_plots] added Campbell River figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,699 INFO [make_plots] added Cherry Point figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,699 INFO [make_plots] added Friday Harbor figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,700 INFO [make_plots] added Halfmoon Bay figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,701 INFO [make_plots] added Nanaimo figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,701 INFO [make_plots] added Neah Bay figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,702 INFO [make_plots] added New Westminster figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,703 INFO [make_plots] added Patricia Bay figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,704 INFO [make_plots] added Point Atkinson figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,705 INFO [make_plots] added Port Renfrew figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,705 INFO [make_plots] added Sand Heads figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,706 INFO [make_plots] added Sandy Cove figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,707 INFO [make_plots] added Squamish figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,708 INFO [make_plots] added Victoria figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,708 INFO [make_plots] added Woodwards Landing figure to 2024-04-19 NEMO forecast publish render list
2024-04-19 10:52:59,709 INFO [make_plots] starting to render nemo forecast publish 19apr24 figures
2024-04-19 10:55:42,784 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17598787 = 2024-04-19 14:04:00 UTC, 58.6% complete
2024-04-19 10:56:25,553 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 198, 30.6% complete
2024-04-19 10:57:03,862 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7619686 = 2024-04-19 15:10:00 UTC, 63.2% complete
2024-04-19 11:00:42,785 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17598968 = 2024-04-19 16:04:40 UTC, 67.0% complete
2024-04-19 11:01:25,554 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 300, 46.3% complete
2024-04-19 11:02:04,346 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7619884 = 2024-04-19 17:22:00 UTC, 72.4% complete
2024-04-19 11:05:42,785 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17599156 = 2024-04-19 18:10:00 UTC, 75.7% complete
2024-04-19 11:06:25,554 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 399, 61.6% complete
2024-04-19 11:07:04,830 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7620082 = 2024-04-19 19:34:00 UTC, 81.6% complete
2024-04-19 11:07:20,710 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/19apr24
2024-04-19 11:10:42,786 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17599341 = 2024-04-19 20:13:20 UTC, 84.3% complete
2024-04-19 11:11:25,555 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete
2024-04-19 11:11:30,083 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/19apr24
2024-04-19 11:11:30,093 INFO [make_surface_current_tiles] finished rendering of tiles for 2024-04-19 forecast into /results/nowcast-sys/figures/surface_currents/forecast/19apr24
2024-04-19 11:11:30,094 INFO [make_surface_current_tiles] surface current tile figures for 2024-04-19 forecast completed
2024-04-19 11:11:30,407 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2024-04-19 11:12:05,307 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7620279 = 2024-04-19 21:45:20 UTC, 90.7% complete
2024-04-19 11:12:08,856 WARNING [make_plots] invalid station number for Boundary Bay station: None
2024-04-19 11:12:08,879 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2024-04-19 11:12:45,731 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:14:40,019 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb952/data for station 07830 Halfmoon Bay from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:14:40,428 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb952 during 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:15:17,699 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:15:42,788 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17599527 = 2024-04-19 22:17:20 UTC, 92.9% complete
2024-04-19 11:16:25,557 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 597, 92.1% complete
2024-04-19 11:16:33,267 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:17:05,791 INFO [watch_NEMO_agrif] 19apr24nowcast-agrif on orcinus-nowcast-agrif: timestep: 7620476 = 2024-04-19 23:56:40 UTC, 99.8% complete
2024-04-19 11:17:11,426 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:17:50,641 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:18:28,288 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:19:06,759 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:19:44,846 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:20:24,035 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94e/data for station 07811 Squamish from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:20:24,442 INFO [make_plots] no obs water level data available from 5cebf1de3d0f4a073c4bb94e during 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:20:42,791 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 17599680 = 2024-04-19 23:59:20 UTC, 100.0% complete
2024-04-19 11:21:01,908 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:21:25,596 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2024-04-19 11:21:25,597 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2024-04-19 11:21:26,036 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:21:27,296 INFO [download_wwatch3_results] running in process 2050407
2024-04-19 11:21:27,297 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:21:27,298 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2024-04-19 11:21:27,299 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:21:40,961 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2024-04-14 00:00:00Z to 2024-04-21 11:55:00Z
2024-04-19 11:22:06,024 INFO [watch_NEMO_agrif] NEMO AGRIF run 11091794.orca2.ibb on orcinus-nowcast-agrif completed
2024-04-19 11:22:06,029 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2024-04-19 11:22:06,417 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:22:07,862 INFO [download_results] running in process 2050414
2024-04-19 11:22:07,863 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:22:07,865 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 11:22:07,866 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:22:07,867 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/19apr24 to /results/SalishSea/nowcast-agrif.201702
2024-04-19 11:22:22,944 INFO [make_plots] finished rendering nemo forecast publish 19apr24 figures
2024-04-19 11:22:22,946 INFO [make_plots] nemo publish plots for 2024-04-19 forecast completed
2024-04-19 11:22:22,955 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-19 11:22:23,178 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:22:31,518 INFO [make_feeds] running in process 2050422
2024-04-19 11:22:31,520 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:22:31,521 INFO [make_feeds] publishing log messages to tcp://*:5567
2024-04-19 11:22:31,522 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:22:40,987 INFO [download_results] nowcast-agrif 2024-04-19 results files from orcinus-nowcast-agrif downloaded
2024-04-19 11:22:40,996 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 11:22:41,217 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:22:45,787 INFO [make_plots] running in process 2050448
2024-04-19 11:22:45,788 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:22:45,789 INFO [make_plots] publishing log messages to tcp://*:5564
2024-04-19 11:22:45,790 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:22:46,408 INFO [make_plots] preparing render list for 2024-04-19 NEMO nowcast-agrif research figures
2024-04-19 11:22:46,409 INFO [make_plots] added baynes_sound_surface figure to 2024-04-19 NEMO nowcast-agrif research render list
2024-04-19 11:22:46,410 INFO [make_plots] starting to render nemo nowcast-agrif research 19apr24 figures
2024-04-19 11:22:57,692 INFO [make_plots] finished rendering nemo nowcast-agrif research 19apr24 figures
2024-04-19 11:22:57,700 INFO [make_plots] nemo research plots for 2024-04-19 nowcast-agrif completed
2024-04-19 11:22:57,729 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-19 11:23:05,037 INFO [make_feeds] ATOM feeds for 2024-04-19 forecast run completed
2024-04-19 11:23:05,040 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2024-04-19 11:24:03,327 INFO [download_wwatch3_results] forecast 2024-04-19 results files from arbutus.cloud-nowcast downloaded
2024-04-19 11:24:03,331 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2024-04-19 11:24:03,521 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:24:04,623 INFO [update_forecast_datasets] running in process 2050488
2024-04-19 11:24:04,625 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:24:04,626 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2024-04-19 11:24:04,627 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:24:04,628 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/19apr24 run
2024-04-19 11:24:04,629 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/19apr24 run
2024-04-19 11:24:04,900 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/19apr24 run: /results/SalishSea/rolling-forecasts/wwatch3
2024-04-19 11:24:04,901 INFO [update_forecast_datasets] wwatch3 2024-04-19 forecast rolling forecast datasets updated
2024-04-19 11:24:04,904 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2024-04-19 11:24:05,084 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2024-04-19 11:24:06,115 INFO [ping_erddap] running in process 2050493
2024-04-19 11:24:06,116 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:24:06,118 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-04-19 11:24:06,119 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:24:06,121 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2024-04-19 11:24:06,123 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-19 11:24:06,309 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:24:10,649 INFO [make_plots] running in process 2050498
2024-04-19 11:24:10,673 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:24:10,691 INFO [make_plots] publishing log messages to tcp://*:5564
2024-04-19 11:24:10,709 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:24:10,713 INFO [make_plots] preparing render list for 2024-04-19 WaveWatch3 forecast2 publish figures
2024-04-19 11:24:10,715 INFO [make_plots] added Halibut Bank figure to 2024-04-19 WaveWatch3 forecast2 publish render list
2024-04-19 11:24:10,716 INFO [make_plots] added Sentry Shoal figure to 2024-04-19 WaveWatch3 forecast2 publish render list
2024-04-19 11:24:10,717 INFO [make_plots] starting to render wwatch3 forecast2 publish 19apr24 figures
2024-04-19 11:24:11,162 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2024-04-19 11:24:11,435 ERROR [make_plots] unexpected TypeError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure
    fig = fig_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/wwatch3/wave_height_period.py", line 69, in make_figure
    _plot_wave_height_time_series(ax_sig_height, buoy, plot_data, theme)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/wwatch3/wave_height_period.py", line 129, in _plot_wave_height_time_series
    plot_data.obs.wave_height.plot(
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/plot/accessor.py", line 47, in __call__
    return dataarray_plot.plot(self._da, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/plot/dataarray_plot.py", line 279, in plot
    raise TypeError("No numeric data to plot.")
TypeError: No numeric data to plot.
2024-04-19 11:24:11,880 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2024-04-19 11:24:11,977 ERROR [make_plots] unexpected TypeError in make_figure:
Traceback (most recent call last):
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1089, in _calc_figure
    fig = fig_func(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/wwatch3/wave_height_period.py", line 69, in make_figure
    _plot_wave_height_time_series(ax_sig_height, buoy, plot_data, theme)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/wwatch3/wave_height_period.py", line 129, in _plot_wave_height_time_series
    plot_data.obs.wave_height.plot(
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/plot/accessor.py", line 47, in __call__
    return dataarray_plot.plot(self._da, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.11/site-packages/xarray/plot/dataarray_plot.py", line 279, in plot
    raise TypeError("No numeric data to plot.")
TypeError: No numeric data to plot.
2024-04-19 11:24:11,992 INFO [make_plots] finished rendering wwatch3 forecast2 publish 19apr24 figures
2024-04-19 11:24:11,993 INFO [make_plots] wwatch3 publish plots for 2024-04-19 forecast2 completed
2024-04-19 11:24:11,995 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-19 11:25:42,799 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2024-04-19 11:25:42,802 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2024-04-19 11:25:43,178 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:25:44,496 INFO [download_results] running in process 2050535
2024-04-19 11:25:44,497 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:25:44,499 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 11:25:44,500 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:25:44,501 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/19apr24 to /results2/SalishSea/nowcast-green.202111
2024-04-19 11:43:20,443 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 11:43:20,917 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-16'], host='localhost')
2024-04-19 11:43:20,922 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 11:43:20,928 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 11:43:22,106 INFO [download_results] running in process 2050579
2024-04-19 11:43:22,107 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:43:22,108 INFO [download_results] publishing log messages to tcp://*:5567
2024-04-19 11:43:22,109 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:43:22,110 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16aug19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 11:43:26,185 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 11:44:25,700 INFO [download_results] hindcast 2019-08-16 results files from optimum-hindcast downloaded
2024-04-19 11:44:25,712 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 11:44:25,901 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-16'], host='localhost')
2024-04-19 11:44:26,971 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 11:59:46,457 INFO [download_results] nowcast-green 2024-04-19 results files from arbutus.cloud-nowcast downloaded
2024-04-19 11:59:46,463 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 11:59:46,654 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:59:46,659 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2024-04-19 11:59:46,663 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:59:46,670 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:59:46,676 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2024-04-19'], host='localhost')
2024-04-19 11:59:47,909 INFO [ping_erddap] running in process 2050689
2024-04-19 11:59:47,910 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:59:47,911 INFO [ping_erddap] publishing log messages to tcp://*:5564
2024-04-19 11:59:47,912 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:59:47,917 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2024-04-19 11:59:47,919 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2024-04-19 11:59:49,991 INFO [make_averaged_dataset] running in process 2050690
2024-04-19 11:59:49,992 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:59:49,993 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2024-04-19 11:59:49,994 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:59:49,995 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Apr-2024 physics
2024-04-19 11:59:49,995 INFO [make_averaged_dataset] running in process 2050691
2024-04-19 11:59:49,996 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:59:49,998 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2024-04-19 11:59:49,999 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:59:50,000 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Apr-2024 chemistry
2024-04-19 11:59:50,005 INFO [make_averaged_dataset] running in process 2050692
2024-04-19 11:59:50,006 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:59:50,007 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567
2024-04-19 11:59:50,009 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:59:50,012 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Apr-2024 biology
2024-04-19 11:59:52,565 INFO [make_plots] running in process 2050687
2024-04-19 11:59:52,566 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 11:59:52,567 INFO [make_plots] publishing log messages to tcp://*:5568
2024-04-19 11:59:52,568 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2024-04-19 11:59:53,209 INFO [make_plots] preparing render list for 2024-04-19 NEMO nowcast-green research figures
2024-04-19 11:59:55,157 INFO [make_plots] added temperature_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 11:59:55,921 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 11:59:56,635 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 11:59:57,413 INFO [make_plots] added silicon_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 11:59:58,165 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 11:59:58,886 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 11:59:59,687 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:00,557 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:01,428 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:02,305 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:03,178 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:03,898 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:04,203 INFO [make_plots] added temperature_salinity_timeseries figure to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:04,204 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:04,204 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:04,205 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2024-04-19 NEMO nowcast-green research render list
2024-04-19 12:00:04,206 INFO [make_plots] starting to render nemo nowcast-green research 19apr24 figures
2024-04-19 12:01:20,141 INFO [make_averaged_dataset] day-averaged dataset for 19-Apr-2024 chemistry created
2024-04-19 12:01:20,145 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-19 12:01:31,318 INFO [make_averaged_dataset] day-averaged dataset for 19-Apr-2024 physics created
2024-04-19 12:01:31,321 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-19 12:02:05,887 INFO [make_averaged_dataset] day-averaged dataset for 19-Apr-2024 biology created
2024-04-19 12:02:05,891 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2024-04-19 13:08:30,770 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 13:08:31,285 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-21'], host='localhost')
2024-04-19 13:08:31,291 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 13:08:31,297 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 13:08:33,051 INFO [download_results] running in process 2051186
2024-04-19 13:08:33,052 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 13:08:33,053 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 13:08:33,054 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 13:08:33,055 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/21aug19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 13:08:37,239 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 13:10:43,392 INFO [download_results] hindcast 2019-08-21 results files from optimum-hindcast downloaded
2024-04-19 13:10:43,402 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 13:10:43,588 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-21'], host='localhost')
2024-04-19 13:10:45,042 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 13:16:18,317 INFO [make_plots] finished rendering nemo nowcast-green research 19apr24 figures
2024-04-19 13:16:18,328 INFO [make_plots] nemo research plots for 2024-04-19 nowcast-green completed
2024-04-19 13:16:18,406 INFO [manager] checklist updated with [plots] items from make_plots worker
2024-04-19 14:53:43,480 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 14:53:43,920 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-08-26'], host='localhost')
2024-04-19 14:53:43,927 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 14:53:43,931 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 14:53:45,316 INFO [download_results] running in process 2051760
2024-04-19 14:53:45,318 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 14:53:45,319 INFO [download_results] publishing log messages to tcp://*:5566
2024-04-19 14:53:45,320 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 14:53:45,321 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/26aug19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 14:53:49,323 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 14:55:34,019 INFO [download_results] hindcast 2019-08-26 results files from optimum-hindcast downloaded
2024-04-19 14:55:34,035 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 14:55:34,258 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-08-26'], host='localhost')
2024-04-19 14:55:35,351 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 14:55:35,570 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['hindcast', '2019-aug', 'robot.graham'], host='localhost')
2024-04-19 14:55:36,552 INFO [archive_tarball] running in process 2051807
2024-04-19 14:55:36,556 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 14:55:36,557 INFO [archive_tarball] publishing log messages to tcp://*:5564
2024-04-19 14:55:36,558 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2024-04-19 14:55:36,559 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-aug19.tar from /ocean/sallen/timbucktwo/oxygen/*aug19/
2024-04-19 15:02:24,579 INFO [archive_tarball] creating /ocean/dlatorne/oxygen-aug19.index from /ocean/dlatorne/oxygen-aug19.tar
2024-04-19 15:02:24,683 INFO [archive_tarball] rsync-ing /ocean/dlatorne/oxygen-aug19.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/oxygen/
2024-04-19 15:10:08,371 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20240419/18/
2024-04-19 15:10:08,372 INFO [collect_weather] 2024-04-19 2.5km weather forecast 18 collection complete
2024-04-19 15:10:08,375 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2024-04-19 15:10:08,776 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2024-04-19 15:10:08,782 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2024-04-19 15:10:08,787 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2024-04-19 15:10:08,791 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2024-04-20'], host='localhost')
2024-04-19 15:10:10,029 INFO [download_weather] running in process 2051876
2024-04-19 15:10:10,035 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 15:10:10,036 INFO [download_weather] publishing log messages to tcp://*:5563
2024-04-19 15:10:10,037 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-19 15:10:10,038 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20240419
2024-04-19 15:10:10,039 INFO [download_weather] running in process 2051874
2024-04-19 15:10:10,040 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 15:10:10,041 INFO [download_weather] publishing log messages to tcp://*:5566
2024-04-19 15:10:10,042 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-19 15:10:10,043 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20240419
2024-04-19 15:10:10,086 INFO [collect_weather] running in process 2051877
2024-04-19 15:10:10,087 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 15:10:10,088 INFO [collect_weather] publishing log messages to tcp://*:5567
2024-04-19 15:10:10,093 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2024-04-19 15:10:10,145 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2024-04-19 15:10:12,597 INFO [crop_gribs] running in process 2051878
2024-04-19 15:10:12,599 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 15:10:12,600 INFO [crop_gribs] publishing log messages to tcp://*:5568
2024-04-19 15:10:12,601 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2024-04-19 15:10:12,602 INFO [crop_gribs] cropping 2024-04-20 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2024-04-19 15:10:12,611 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20240420/00/
2024-04-19 15:13:58,227 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20240419/18/
2024-04-19 15:13:58,228 INFO [crop_gribs] 2024-04-19 18 GRIBs cropping complete
2024-04-19 15:13:58,231 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2024-04-19 15:16:05,249 INFO [download_weather] 2024-04-19 1km weather forecast 12 downloads complete
2024-04-19 15:16:05,253 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-19 15:16:45,697 INFO [download_weather] 2024-04-19 1km weather forecast 00 downloads complete
2024-04-19 15:16:45,700 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2024-04-19 15:55:27,947 INFO [archive_tarball] hindcast *aug19 results files archived to robot.graham
2024-04-19 15:55:27,952 INFO [manager] checklist updated with [archived tarballs] items from archive_tarball worker
2024-04-19 16:18:54,222 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 16:18:54,648 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-01'], host='localhost')
2024-04-19 16:18:54,653 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 16:18:54,658 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 16:18:56,401 INFO [download_results] running in process 2052574
2024-04-19 16:18:56,403 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 16:18:56,405 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 16:18:56,406 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 16:18:56,407 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/01sep19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 16:19:00,457 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 16:19:57,383 INFO [download_results] hindcast 2019-09-01 results files from optimum-hindcast downloaded
2024-04-19 16:19:57,400 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 16:19:57,616 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-01'], host='localhost')
2024-04-19 16:19:58,914 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 17:44:04,898 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 17:44:05,301 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-06'], host='localhost')
2024-04-19 17:44:05,309 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 17:44:05,313 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 17:44:07,217 INFO [download_results] running in process 2053438
2024-04-19 17:44:07,218 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 17:44:07,219 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 17:44:07,221 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 17:44:07,222 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/06sep19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 17:44:11,273 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 17:45:10,234 INFO [download_results] hindcast 2019-09-06 results files from optimum-hindcast downloaded
2024-04-19 17:45:10,250 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 17:45:10,487 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-06'], host='localhost')
2024-04-19 17:45:11,566 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 19:09:15,798 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 19:09:16,220 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-11'], host='localhost')
2024-04-19 19:09:16,226 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 19:09:16,231 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 19:09:17,768 INFO [download_results] running in process 2053775
2024-04-19 19:09:17,770 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 19:09:17,771 INFO [download_results] publishing log messages to tcp://*:5564
2024-04-19 19:09:17,772 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 19:09:17,774 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/11sep19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 19:09:21,854 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 19:10:25,190 INFO [download_results] hindcast 2019-09-11 results files from optimum-hindcast downloaded
2024-04-19 19:10:25,210 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 19:10:25,453 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-11'], host='localhost')
2024-04-19 19:10:26,540 INFO [manager] checklist updated with [results splitting] items from split_results worker
2024-04-19 20:39:26,565 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_hindcast worker
2024-04-19 20:39:26,991 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['optimum-hindcast', 'hindcast', '--run-date', '2019-09-16'], host='localhost')
2024-04-19 20:39:26,996 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 20:39:27,001 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_hindcast', args=['optimum-hindcast'], host='localhost')
2024-04-19 20:39:28,485 INFO [download_results] running in process 2054209
2024-04-19 20:39:28,486 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2024-04-19 20:39:28,487 INFO [download_results] publishing log messages to tcp://*:5562
2024-04-19 20:39:28,488 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2024-04-19 20:39:28,489 INFO [download_results] downloading results from optimum-hindcast:/scratch/sallen/dlatorne/oxygen/16sep19 to /ocean/sallen/timbucktwo/oxygen
2024-04-19 20:39:32,599 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_hindcast worker
2024-04-19 20:40:30,977 INFO [download_results] hindcast 2019-09-16 results files from optimum-hindcast downloaded
2024-04-19 20:40:31,010 INFO [manager] checklist updated with [results files] items from download_results worker
2024-04-19 20:40:31,274 INFO [manager] launching NextWorker(module='nowcast.workers.split_results', args=['hindcast', '2019-09-16'], host='localhost')
2024-04-19 20:40:32,517 INFO [manager] checklist updated with [results splitting] items from split_results worker