2025-07-10 04:00:26,043 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-07-10 04:00:26,049 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-07-10 04:00:26,055 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-07-10 04:00:26,055 INFO [rotate_logs] log files rotated
2025-07-10 04:00:26,060 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-07-10 04:02:23,289 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/09jul25
2025-07-10 04:02:23,312 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-07-09 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/09jul25
2025-07-10 04:02:23,313 INFO [make_surface_current_tiles] surface current tile figures for 2025-07-09 forecast2 completed
2025-07-10 04:02:23,790 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-07-10 04:03:22,077 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 414, 76.7% complete
2025-07-10 04:08:22,088 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 516, 95.6% complete
2025-07-10 04:13:22,086 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-10 04:13:22,144 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-10 04:13:22,428 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 04:13:23,760 INFO [download_wwatch3_results] running in process 1457136
2025-07-10 04:13:23,762 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 04:13:23,763 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-07-10 04:13:23,764 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-10 04:14:02,391 INFO [download_wwatch3_results] forecast2 2025-07-10 results files from arbutus.cloud-nowcast downloaded
2025-07-10 04:14:02,395 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-10 04:14:02,553 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 04:14:03,737 INFO [update_forecast_datasets] running in process 1457149
2025-07-10 04:14:03,739 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 04:14:03,740 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-07-10 04:14:03,741 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-10 04:14:03,742 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/10jul25 run
2025-07-10 04:14:03,839 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/10jul25 run
2025-07-10 04:14:03,840 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/10jul25 run
2025-07-10 04:14:28,519 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/10jul25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-07-10 04:14:28,520 INFO [update_forecast_datasets] wwatch3 2025-07-10 forecast2 rolling forecast datasets updated
2025-07-10 04:14:28,523 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-10 04:14:28,677 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-07-10 04:14:29,940 INFO [ping_erddap] running in process 1457165
2025-07-10 04:14:29,941 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 04:14:29,942 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-10 04:14:29,943 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-10 04:14:29,945 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-07-10 04:14:29,947 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-10 04:14:30,117 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 04:14:33,345 INFO [make_plots] running in process 1457170
2025-07-10 04:14:33,347 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 04:14:33,350 INFO [make_plots] publishing log messages to tcp://*:5563
2025-07-10 04:14:33,351 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-10 04:14:33,351 INFO [make_plots] preparing render list for 2025-07-10 WaveWatch3 forecast2 publish figures
2025-07-10 04:14:33,352 INFO [make_plots] added Halibut Bank figure to 2025-07-10 WaveWatch3 forecast2 publish render list
2025-07-10 04:14:33,353 INFO [make_plots] added Sentry Shoal figure to 2025-07-10 WaveWatch3 forecast2 publish render list
2025-07-10 04:14:33,354 INFO [make_plots] starting to render wwatch3 forecast2 publish 10jul25 figures
2025-07-10 04:14:33,796 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-07-10 04:14:39,750 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-07-10 04:14:43,981 INFO [make_plots] finished rendering wwatch3 forecast2 publish 10jul25 figures
2025-07-10 04:14:43,982 INFO [make_plots] wwatch3 publish plots for 2025-07-10 forecast2 completed
2025-07-10 04:14:43,986 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-10 09:05:35,950 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250710/12/
2025-07-10 09:05:35,951 INFO [collect_weather] 2025-07-10 2.5km weather forecast 12 collection complete
2025-07-10 09:05:35,954 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-10 09:05:36,134 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250710/12/
2025-07-10 09:05:36,135 INFO [crop_gribs] 2025-07-10 12 GRIBs cropping complete
2025-07-10 09:05:36,309 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-07-09'], host='localhost')
2025-07-10 09:05:36,317 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-07-09'], host='localhost')
2025-07-10 09:05:36,322 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-07-09'], host='localhost')
2025-07-10 09:05:36,326 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-07-09'], host='localhost')
2025-07-10 09:05:36,331 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-07-10 09:05:36,337 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-07-10 09:05:36,341 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-07-10 09:05:36,347 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-07-10 09:05:36,353 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-07-10 09:05:36,360 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-10 09:05:36,841 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-07-10 09:05:37,992 INFO [collect_NeahBay_ssh] running in process 1460766
2025-07-10 09:05:37,993 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:37,993 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-07-10 09:05:37,994 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:37,995 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-07-10
2025-07-10 09:05:38,032 INFO [collect_weather] running in process 1460768
2025-07-10 09:05:38,033 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,033 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-07-10 09:05:38,034 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,163 INFO [download_live_ocean] running in process 1460767
2025-07-10 09:05:38,164 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,164 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-07-10 09:05:38,165 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,166 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-07-10
2025-07-10 09:05:38,224 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-07-10 09:05:38,510 INFO [collect_river_data] running in process 1460762
2025-07-10 09:05:38,511 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,511 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-07-10 09:05:38,512 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,513 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-07-09
2025-07-10 09:05:38,514 INFO [collect_river_data] running in process 1460763
2025-07-10 09:05:38,515 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,516 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-07-10 09:05:38,517 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,522 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-07-09
2025-07-10 09:05:38,559 INFO [collect_river_data] running in process 1460764
2025-07-10 09:05:38,560 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,561 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-07-10 09:05:38,562 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,563 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-07-09
2025-07-10 09:05:38,639 INFO [collect_river_data] running in process 1460760
2025-07-10 09:05:38,640 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,641 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-07-10 09:05:38,642 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,643 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-07-09
2025-07-10 09:05:38,653 INFO [make_turbidity_file] running in process 1460765
2025-07-10 09:05:38,654 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,655 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-07-10 09:05:38,657 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,658 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-07-10
2025-07-10 09:05:38,840 INFO [crop_gribs] running in process 1460769
2025-07-10 09:05:38,841 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,842 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-07-10 09:05:38,843 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,844 INFO [crop_gribs] cropping 2025-07-10 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-07-10 09:05:38,859 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250710/18/
2025-07-10 09:05:38,895 INFO [grib_to_netcdf] running in process 1460792
2025-07-10 09:05:38,896 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:38,897 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2025-07-10 09:05:38,898 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:38,899 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-07-10 nowcast and forecast runs
2025-07-10 09:05:38,995 INFO [download_live_ocean] downloaded 10215956 bytes from https://liveocean.apl.uw.edu/output/f2025.07.10/ubc.nc to /results/forcing/LiveOcean/downloaded/20250710/low_passed_UBC.nc
2025-07-10 09:05:38,996 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-07-10 09:05:39,014 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-07-09&endDT=2025-07-09 "HTTP/1.1 200 "
2025-07-10 09:05:39,020 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-07-09 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-07-10 09:05:39,021 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-07-09 completed
2025-07-10 09:05:39,022 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-10 09:05:39,094 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-07-09&endDT=2025-07-09 "HTTP/1.1 200 "
2025-07-10 09:05:39,103 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-07-09&endDT=2025-07-09 "HTTP/1.1 200 "
2025-07-10 09:05:39,116 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-07-09&endDT=2025-07-09 "HTTP/1.1 200 "
2025-07-10 09:05:39,120 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-07-09 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-07-10 09:05:39,121 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-07-09 completed
2025-07-10 09:05:39,125 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-07-09 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-07-10 09:05:39,126 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-07-09 completed
2025-07-10 09:05:39,159 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-07-09 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-07-10 09:05:39,161 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-07-09 completed
2025-07-10 09:05:39,183 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-10 09:05:39,341 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-10 09:05:39,490 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-10 09:05:40,016 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250710/low_passed_UBC.nc
2025-07-10 09:05:40,017 INFO [download_live_ocean] 2025-07-10 Live Ocean file for Salish Sea western boundary downloaded
2025-07-10 09:05:40,020 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-07-10 09:05:40,131 INFO [collect_NeahBay_ssh] 2025-07-10T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-07-10 09:05:40,344 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-07-10'], host='localhost')
2025-07-10 09:05:40,352 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-07-10 09:05:40,539 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 09:05:41,424 INFO [make_turbidity_file] 2025-07-10 Fraser River turbidity file creation complete
2025-07-10 09:05:41,433 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-07-10 09:05:42,500 INFO [make_live_ocean_files] running in process 1460900
2025-07-10 09:05:42,501 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:42,503 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2025-07-10 09:05:42,504 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:42,505 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-07-10 Live Ocean run
2025-07-10 09:05:43,256 INFO [make_ssh_files] running in process 1460902
2025-07-10 09:05:43,257 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:43,258 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-07-10 09:05:43,259 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:43,261 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-07-10 NOAA Neah Bay 06Z observation and forecast values
2025-07-10 09:05:44,977 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d06.nc
2025-07-10 09:05:45,165 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d07.nc
2025-07-10 09:05:45,349 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d08.nc
2025-07-10 09:05:45,529 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d09.nc
2025-07-10 09:05:45,737 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d10.nc
2025-07-10 09:05:45,928 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d11.nc
2025-07-10 09:05:46,124 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d12.nc
2025-07-10 09:05:46,305 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d13.nc
2025-07-10 09:05:46,419 INFO [make_ssh_files] sea surface height boundary file for 2025-07-10 nowcast run created
2025-07-10 09:05:46,422 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-07-10 09:05:46,582 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-07-10 09:05:46,586 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-07-10 09:05:47,900 INFO [make_201702_runoff_file] running in process 1460946
2025-07-10 09:05:47,913 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:47,914 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-07-10 09:05:47,915 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:47,945 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-07-10 09:05:48,037 INFO [make_201702_runoff_file] 2025-07-10 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-07-10 09:05:48,040 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-07-10 09:05:48,567 INFO [make_runoff_file] running in process 1460944
2025-07-10 09:05:48,568 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:05:48,569 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-07-10 09:05:48,570 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:05:48,571 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-07-09
2025-07-10 09:05:50,161 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m07d10.nc
2025-07-10 09:05:51,431 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-07-09: /results/forcing/rivers/R202108Dailies_y2025m07d09.nc
2025-07-10 09:05:51,434 INFO [make_runoff_file] 2025-07-09 runoff file creation completed
2025-07-10 09:05:51,435 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-07-10 09:05:57,683 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d11.nc
2025-07-10 09:06:01,956 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d12.nc
2025-07-10 09:06:01,970 INFO [grib_to_netcdf] 2025-07-10 NEMO-atmos forcing file for nowcast+ created
2025-07-10 09:06:01,973 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-07-10 09:06:24,183 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d10.nc
2025-07-10 09:06:24,184 INFO [make_live_ocean_files] 2025-07-10 Live Ocean western boundary conditions files created
2025-07-10 09:06:24,186 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-07-10 09:06:24,481 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-07-10 09:06:24,483 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-07-10 09:06:24,484 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-07-10 09:06:24,485 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-07-10 09:06:24,486 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-07-10 09:06:25,409 INFO [ping_erddap] running in process 1460991
2025-07-10 09:06:25,411 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:06:25,412 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-07-10 09:06:25,413 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:06:25,414 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-07-10 09:06:25,416 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-10 09:06:25,474 INFO [upload_forcing] running in process 1460993
2025-07-10 09:06:25,475 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:06:25,476 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-10 09:06:25,478 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:06:25,499 INFO [upload_forcing] running in process 1460992
2025-07-10 09:06:25,500 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:06:25,502 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-07-10 09:06:25,503 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:06:25,504 INFO [upload_forcing] running in process 1460995
2025-07-10 09:06:25,506 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:06:25,507 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-10 09:06:25,507 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:06:25,542 INFO [upload_forcing] running in process 1460994
2025-07-10 09:06:25,543 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:06:25,544 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-07-10 09:06:25,545 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:06:27,707 INFO [upload_forcing] nowcast+ 2025-07-10 forcing files upload to optimum-hindcast completed
2025-07-10 09:06:27,711 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-10 09:06:27,803 INFO [upload_forcing] nowcast+ 2025-07-10 forcing files upload to orcinus-nowcast-agrif completed
2025-07-10 09:06:27,874 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-10 09:06:34,408 INFO [upload_forcing] nowcast+ 2025-07-10 forcing files upload to arbutus.cloud-nowcast completed
2025-07-10 09:06:34,414 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-10 09:06:34,571 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 09:06:35,753 INFO [make_forcing_links] running in process 1461034
2025-07-10 09:06:35,754 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:06:35,755 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-07-10 09:06:35,756 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:06:36,475 INFO [upload_forcing] nowcast+ 2025-07-10 forcing files upload to robot.graham completed
2025-07-10 09:06:36,480 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-10 09:06:36,957 INFO [make_forcing_links] nowcast+ 2025-07-10 forcing file links on arbutus.cloud-nowcast created
2025-07-10 09:06:36,961 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-10 09:06:37,120 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-10'], host='arbutus.cloud-nowcast')
2025-07-10 09:06:39,220 INFO [run_NEMO] running in process 13625
2025-07-10 09:06:39,221 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:06:39,223 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-10 09:06:39,228 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:06:39,916 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-10 09:06:40,223 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/10jul25nowcast_2025-07-10T090639.266288-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-10 09:06:40,236 INFO [run_NEMO] nowcast NEMO run for 2025-07-10 on arbutus.cloud-nowcast started
2025-07-10 09:06:40,240 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-10 09:06:40,402 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-07-10 09:06:42,336 INFO [watch_NEMO] running in process 13898
2025-07-10 09:06:42,338 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:06:42,339 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-10 09:06:42,340 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:06:42,410 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-10 09:11:42,412 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18563504 = 2025-07-10 05:08:40 UTC, 21.4% complete
2025-07-10 09:16:42,413 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18563984 = 2025-07-10 10:28:40 UTC, 43.7% complete
2025-07-10 09:21:42,411 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18564479 = 2025-07-10 15:58:40 UTC, 66.6% complete
2025-07-10 09:26:42,413 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18564949 = 2025-07-10 21:12:00 UTC, 88.4% complete
2025-07-10 09:31:42,416 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-07-10 09:31:42,419 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-10 09:31:42,771 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 09:31:42,778 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 09:31:43,855 INFO [download_results] running in process 1461338
2025-07-10 09:31:43,856 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:31:43,857 INFO [download_results] publishing log messages to tcp://*:5562
2025-07-10 09:31:43,858 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:31:43,858 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/10jul25 to /results/SalishSea/nowcast-blue.202111
2025-07-10 09:31:43,881 INFO [make_forcing_links] running in process 1461337
2025-07-10 09:31:43,882 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:31:43,883 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-07-10 09:31:43,884 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:31:44,895 INFO [make_forcing_links] ssh 2025-07-10 forcing file links on arbutus.cloud-nowcast created
2025-07-10 09:31:44,899 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-10 09:31:45,061 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-10'], host='arbutus.cloud-nowcast')
2025-07-10 09:31:47,131 INFO [run_NEMO] running in process 14284
2025-07-10 09:31:47,132 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:31:47,133 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-10 09:31:47,134 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:31:47,872 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-10 09:31:48,181 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/10jul25forecast_2025-07-10T093147.204128-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-10 09:31:48,194 INFO [run_NEMO] forecast NEMO run for 2025-07-10 on arbutus.cloud-nowcast started
2025-07-10 09:31:48,198 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-10 09:31:48,358 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-07-10 09:31:51,520 INFO [watch_NEMO] running in process 14560
2025-07-10 09:31:51,522 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:31:51,523 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-10 09:31:51,524 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:31:51,597 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-10 09:36:51,597 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18565658 = 2025-07-11 05:04:40 UTC, 14.1% complete
2025-07-10 09:41:51,597 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18566138 = 2025-07-11 10:24:40 UTC, 28.9% complete
2025-07-10 09:46:51,598 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18566616 = 2025-07-11 15:43:20 UTC, 43.7% complete
2025-07-10 09:49:20,253 INFO [download_results] nowcast 2025-07-10 results files from arbutus.cloud-nowcast downloaded
2025-07-10 09:49:20,258 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-10 09:49:20,423 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 09:49:20,429 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-07-09'], host='localhost')
2025-07-10 09:49:20,434 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 09:49:22,369 INFO [make_CHS_currents_file] running in process 1461458
2025-07-10 09:49:22,371 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:49:22,371 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-07-10 09:49:22,372 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:49:23,456 INFO [make_plots] running in process 1461455
2025-07-10 09:49:23,458 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:49:23,458 INFO [make_plots] publishing log messages to tcp://*:5564
2025-07-10 09:49:23,459 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:49:23,489 INFO [make_plots] running in process 1461457
2025-07-10 09:49:23,500 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 09:49:23,507 INFO [make_plots] publishing log messages to tcp://*:5565
2025-07-10 09:49:23,508 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-10 09:49:23,819 INFO [make_plots] preparing render list for 2025-07-10 NEMO nowcast-blue research figures
2025-07-10 09:49:23,846 INFO [make_plots] preparing render list for 2025-07-09 NEMO nowcast-blue comparison figures
2025-07-10 09:49:23,955 INFO [make_plots] added SH_wind figure to 2025-07-09 NEMO nowcast-blue comparison render list
2025-07-10 09:49:23,956 INFO [make_plots] added Compare_VENUS_East figure to 2025-07-09 NEMO nowcast-blue comparison render list
2025-07-10 09:49:23,957 INFO [make_plots] added Compare_VENUS_Central figure to 2025-07-09 NEMO nowcast-blue comparison render list
2025-07-10 09:49:23,958 INFO [make_plots] starting to render nemo nowcast comparison 09jul25 figures
2025-07-10 09:49:25,293 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-blue research render list
2025-07-10 09:49:26,417 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-blue research render list
2025-07-10 09:49:26,417 INFO [make_plots] added Currents_sections_and_surface figure to 2025-07-10 NEMO nowcast-blue research render list
2025-07-10 09:49:26,418 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-07-10 NEMO nowcast-blue research render list
2025-07-10 09:49:26,419 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-07-10 NEMO nowcast-blue research render list
2025-07-10 09:49:26,419 INFO [make_plots] starting to render nemo nowcast research 10jul25 figures
2025-07-10 09:49:56,785 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-10 for nowcast
2025-07-10 09:49:56,788 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-10 09:50:37,188 INFO [make_plots] finished rendering nemo nowcast research 10jul25 figures
2025-07-10 09:50:37,190 INFO [make_plots] nemo research plots for 2025-07-10 nowcast completed
2025-07-10 09:50:37,210 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-10 09:50:57,146 INFO [make_plots] finished rendering nemo nowcast comparison 09jul25 figures
2025-07-10 09:50:57,149 INFO [make_plots] nemo comparison plots for 2025-07-09 nowcast completed
2025-07-10 09:50:57,153 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-10 09:51:51,608 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18567091 = 2025-07-11 21:00:00 UTC, 58.4% complete
2025-07-10 09:56:51,779 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18567554 = 2025-07-12 02:08:40 UTC, 72.6% complete
2025-07-10 10:01:51,626 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18568029 = 2025-07-12 07:25:20 UTC, 87.3% complete
2025-07-10 10:06:51,637 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18568440 = 2025-07-12 11:59:20 UTC, 100.0% complete
2025-07-10 10:11:51,627 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-07-10 10:11:51,634 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-10 10:11:51,999 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-10'], host='arbutus.cloud-nowcast')
2025-07-10 10:11:52,007 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-10'], host='arbutus.cloud-nowcast')
2025-07-10 10:11:52,012 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:11:52,020 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:11:52,024 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:11:52,030 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:11:52,036 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:11:53,129 INFO [download_results] running in process 1461861
2025-07-10 10:11:53,131 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:53,132 INFO [download_results] publishing log messages to tcp://*:5562
2025-07-10 10:11:53,133 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:53,134 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/10jul25 to /results/SalishSea/forecast.202111
2025-07-10 10:11:53,166 INFO [upload_forcing] running in process 1461858
2025-07-10 10:11:53,167 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:53,168 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-10 10:11:53,169 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:53,174 INFO [upload_forcing] running in process 1461860
2025-07-10 10:11:53,175 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:53,176 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-07-10 10:11:53,177 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:53,178 INFO [upload_forcing] running in process 1461857
2025-07-10 10:11:53,179 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:53,180 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-10 10:11:53,181 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:53,182 INFO [upload_forcing] running in process 1461859
2025-07-10 10:11:53,183 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:53,183 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-07-10 10:11:53,184 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:53,835 INFO [upload_forcing] turbidity 2025-07-10 forcing files upload to optimum-hindcast completed
2025-07-10 10:11:53,839 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-10 10:11:53,923 INFO [upload_forcing] turbidity 2025-07-10 forcing files upload to orcinus-nowcast-agrif completed
2025-07-10 10:11:54,005 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-10 10:11:54,165 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:11:54,728 INFO [make_ww3_wind_file] running in process 15241
2025-07-10 10:11:54,729 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:54,729 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-07-10 10:11:54,789 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:54,790 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-07-10 forecast run
2025-07-10 10:11:54,834 INFO [make_ww3_current_file] running in process 15240
2025-07-10 10:11:54,835 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:54,836 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-07-10 10:11:54,837 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:54,838 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-07-10 forecast run
2025-07-10 10:11:55,191 INFO [make_forcing_links] running in process 1461893
2025-07-10 10:11:55,192 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:55,193 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-07-10 10:11:55,194 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:55,963 INFO [make_forcing_links] nowcast-agrif 2025-07-10 forcing file links on orcinus-nowcast-agrif created
2025-07-10 10:11:55,967 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-10 10:11:56,132 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:11:57,136 INFO [run_NEMO_agrif] running in process 1461902
2025-07-10 10:11:57,137 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:57,138 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2025-07-10 10:11:57,139 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:11:58,083 INFO [upload_forcing] turbidity 2025-07-10 forcing files upload to arbutus.cloud-nowcast completed
2025-07-10 10:11:58,087 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-10 10:11:58,250 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:11:58,656 INFO [upload_forcing] turbidity 2025-07-10 forcing files upload to robot.graham completed
2025-07-10 10:11:58,660 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-10 10:11:59,284 INFO [make_forcing_links] running in process 1461916
2025-07-10 10:11:59,286 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:11:59,287 INFO [make_forcing_links] publishing log messages to tcp://*:5565
2025-07-10 10:11:59,287 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:12:05,982 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-07-10 forecast run
2025-07-10 10:12:06,240 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-07-10 10:12:10,010 INFO [make_forcing_links] nowcast-green 2025-07-10 forcing file links on arbutus.cloud-nowcast created
2025-07-10 10:12:10,015 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-10 10:12:10,185 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-10'], host='arbutus.cloud-nowcast')
2025-07-10 10:12:13,872 INFO [run_NEMO] running in process 15880
2025-07-10 10:12:13,873 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:12:13,874 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-10 10:12:13,875 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:12:14,711 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-10 10:12:15,032 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/10jul25nowcast-green_2025-07-10T101213.920568-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-10 10:12:15,046 INFO [run_NEMO] nowcast-green NEMO run for 2025-07-10 on arbutus.cloud-nowcast started
2025-07-10 10:12:15,051 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-10 10:12:15,223 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-07-10 10:12:17,927 INFO [watch_NEMO] running in process 16156
2025-07-10 10:12:17,929 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:12:17,930 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-10 10:12:17,931 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:12:18,259 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-10 10:12:18,833 INFO [run_NEMO_agrif] job id for 10jul25nowcast-agrif: 11442412.orca2.ibb
2025-07-10 10:12:18,834 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2025-07-10 queued on orcinus-nowcast-agrif
2025-07-10 10:12:18,839 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2025-07-10 10:12:19,011 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11442412.orca2.ibb'], host='localhost')
2025-07-10 10:12:20,038 INFO [watch_NEMO_agrif] running in process 1461927
2025-07-10 10:12:20,040 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:12:20,041 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5564
2025-07-10 10:12:20,043 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:12:20,848 INFO [watch_NEMO_agrif] watching 10jul25nowcast-agrif job 11442412 on orcinus-nowcast-agrif
2025-07-10 10:12:21,080 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif job 11442412 is queued on orcinus-nowcast-agrif
2025-07-10 10:13:21,285 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif job 11442412 is queued on orcinus-nowcast-agrif
2025-07-10 10:14:22,314 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2025-07-10 10:14:26,025 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-07-10 forecast run
2025-07-10 10:14:26,044 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-07-10 10:14:26,214 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-10'], host='arbutus.cloud-nowcast')
2025-07-10 10:14:28,975 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-10 10:14:29,147 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-07-10 10:14:31,499 INFO [watch_ww3] running in process 16348
2025-07-10 10:14:31,500 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:14:31,501 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-07-10 10:14:31,503 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:14:31,597 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-10 10:17:18,256 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18563207 = 2025-07-10 01:50:40 UTC, 7.7% complete
2025-07-10 10:19:22,790 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8584038 = 2025-07-10 02:11:20 UTC, 9.1% complete
2025-07-10 10:19:31,595 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 93, 21.5% complete
2025-07-10 10:20:19,945 INFO [download_results] forecast 2025-07-10 results files from arbutus.cloud-nowcast downloaded
2025-07-10 10:20:19,950 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-10 10:20:20,117 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:20:22,346 INFO [make_CHS_currents_file] running in process 1461986
2025-07-10 10:20:22,348 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:20:22,349 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-07-10 10:20:22,350 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:21:10,148 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-10 for forecast
2025-07-10 10:21:10,151 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-10 10:21:10,320 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:21:11,496 INFO [update_forecast_datasets] running in process 1462007
2025-07-10 10:21:11,498 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:21:11,499 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-07-10 10:21:11,500 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:21:11,501 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/10jul25 run
2025-07-10 10:21:11,608 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/10jul25 run: /results/SalishSea/rolling-forecasts/nemo
2025-07-10 10:21:11,609 INFO [update_forecast_datasets] nemo 2025-07-10 forecast rolling forecast datasets updated
2025-07-10 10:21:11,611 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-10 10:21:11,779 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-07-10 10:21:11,784 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:21:11,789 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:21:12,962 INFO [ping_erddap] running in process 1462012
2025-07-10 10:21:12,963 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:21:12,964 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-07-10 10:21:12,964 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:21:12,973 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-07-10 10:21:12,975 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-10 10:21:14,167 INFO [make_surface_current_tiles] running in process 1462015
2025-07-10 10:21:14,168 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:21:14,169 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-07-10 10:21:14,170 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:21:14,172 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-07-10 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/10jul25
2025-07-10 10:21:15,103 INFO [make_plots] running in process 1462014
2025-07-10 10:21:15,104 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:21:15,105 INFO [make_plots] publishing log messages to tcp://*:5565
2025-07-10 10:21:15,111 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:21:15,590 INFO [make_plots] preparing render list for 2025-07-10 NEMO forecast publish figures
2025-07-10 10:21:15,676 INFO [make_plots] added Website_thumbnail figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,676 INFO [make_plots] added Threshold_website figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,677 INFO [make_plots] added PA_tidal_predictions figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,678 INFO [make_plots] added Boundary Bay figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,678 INFO [make_plots] added Campbell River figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,679 INFO [make_plots] added Cherry Point figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,679 INFO [make_plots] added Friday Harbor figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,680 INFO [make_plots] added Halfmoon Bay figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,680 INFO [make_plots] added Nanaimo figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,681 INFO [make_plots] added Neah Bay figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,682 INFO [make_plots] added New Westminster figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,682 INFO [make_plots] added Patricia Bay figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,683 INFO [make_plots] added Point Atkinson figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,684 INFO [make_plots] added Port Renfrew figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,684 INFO [make_plots] added Sand Heads figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,685 INFO [make_plots] added Sandy Cove figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,838 INFO [make_plots] added Squamish figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,845 INFO [make_plots] added Victoria figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,846 INFO [make_plots] added Woodwards Landing figure to 2025-07-10 NEMO forecast publish render list
2025-07-10 10:21:15,847 INFO [make_plots] starting to render nemo forecast publish 10jul25 figures
2025-07-10 10:22:18,256 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18563393 = 2025-07-10 03:54:40 UTC, 16.3% complete
2025-07-10 10:24:23,231 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8584289 = 2025-07-10 04:58:40 UTC, 20.8% complete
2025-07-10 10:24:31,597 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 192, 44.4% complete
2025-07-10 10:27:18,263 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18563578 = 2025-07-10 05:58:00 UTC, 24.9% complete
2025-07-10 10:29:03,334 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-07-10 10:29:03,335 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-07-10 10:29:23,713 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8584541 = 2025-07-10 07:46:40 UTC, 32.4% complete
2025-07-10 10:29:29,386 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:29:31,596 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 297, 68.8% complete
2025-07-10 10:30:34,681 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-07-10 10:30:34,682 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-07-10 10:30:55,898 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:31:38,372 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:31:54,639 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/10jul25
2025-07-10 10:31:56,906 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:32:15,353 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:32:18,269 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18563765 = 2025-07-10 08:02:40 UTC, 33.5% complete
2025-07-10 10:32:33,535 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:32:49,877 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:33:08,073 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:33:25,940 WARNING [make_plots] invalid station number for Squamish station: None
2025-07-10 10:33:25,940 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-07-10 10:33:43,247 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:33:59,742 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-07-05 00:00:00Z to 2025-07-12 11:55:00Z
2025-07-10 10:34:17,466 INFO [make_plots] finished rendering nemo forecast publish 10jul25 figures
2025-07-10 10:34:17,469 INFO [make_plots] nemo publish plots for 2025-07-10 forecast completed
2025-07-10 10:34:17,476 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-10 10:34:17,640 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:34:20,633 INFO [make_feeds] running in process 1463859
2025-07-10 10:34:20,634 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:34:20,634 INFO [make_feeds] publishing log messages to tcp://*:5565
2025-07-10 10:34:20,635 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:34:24,147 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8584793 = 2025-07-10 10:34:40 UTC, 44.1% complete
2025-07-10 10:34:31,645 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 396, 91.7% complete
2025-07-10 10:34:35,939 INFO [make_feeds] ATOM feeds for 2025-07-10 forecast run completed
2025-07-10 10:34:35,942 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-07-10 10:36:36,973 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/10jul25
2025-07-10 10:36:36,997 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-07-10 forecast into /results/nowcast-sys/figures/surface_currents/forecast/10jul25
2025-07-10 10:36:36,998 INFO [make_surface_current_tiles] surface current tile figures for 2025-07-10 forecast completed
2025-07-10 10:36:37,492 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-07-10 10:37:18,263 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18563955 = 2025-07-10 10:09:20 UTC, 42.3% complete
2025-07-10 10:39:24,582 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8585044 = 2025-07-10 13:22:00 UTC, 55.7% complete
2025-07-10 10:39:31,658 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-10 10:39:31,936 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-10 10:39:32,140 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 10:39:32,146 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-10'], host='arbutus.cloud-nowcast')
2025-07-10 10:39:33,171 INFO [download_wwatch3_results] running in process 1463964
2025-07-10 10:39:33,172 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:39:33,173 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-07-10 10:39:33,174 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:39:34,840 INFO [run_ww3] running in process 16629
2025-07-10 10:39:34,841 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 10:39:34,842 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-07-10 10:39:34,850 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-10 10:39:34,856 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-07-10T103934.825374-0700
2025-07-10 10:39:34,864 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-07-10T103934.825374-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-07-10 10:39:34,867 INFO [run_ww3] forecast WaveWatch3 run for 2025-07-10 on arbutus.cloud-nowcast started
2025-07-10 10:39:34,993 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-10 10:39:35,306 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-07-10 10:41:48,104 INFO [download_wwatch3_results] nowcast 2025-07-10 results files from arbutus.cloud-nowcast downloaded
2025-07-10 10:41:48,108 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-10 10:42:18,259 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18564140 = 2025-07-10 12:12:40 UTC, 50.9% complete
2025-07-10 10:44:25,022 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8585295 = 2025-07-10 16:09:20 UTC, 67.3% complete
2025-07-10 10:44:38,165 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 90, 13.9% complete
2025-07-10 10:47:18,276 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18564327 = 2025-07-10 14:17:20 UTC, 59.6% complete
2025-07-10 10:49:25,460 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8585549 = 2025-07-10 18:58:40 UTC, 79.1% complete
2025-07-10 10:49:38,151 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 192, 29.6% complete
2025-07-10 10:52:18,264 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18564515 = 2025-07-10 16:22:40 UTC, 68.3% complete
2025-07-10 10:54:25,901 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8585800 = 2025-07-10 21:46:00 UTC, 90.7% complete
2025-07-10 10:54:38,164 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 297, 45.8% complete
2025-07-10 10:57:18,294 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18564700 = 2025-07-10 18:26:00 UTC, 76.8% complete
2025-07-10 10:59:26,399 INFO [watch_NEMO_agrif] 10jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8586000 = 2025-07-10 23:59:20 UTC, 100.0% complete
2025-07-10 10:59:38,126 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 396, 61.1% complete
2025-07-10 11:02:18,364 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18564883 = 2025-07-10 20:28:00 UTC, 85.3% complete
2025-07-10 11:04:26,620 INFO [watch_NEMO_agrif] NEMO AGRIF run 11442412.orca2.ibb on orcinus-nowcast-agrif completed
2025-07-10 11:04:26,625 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2025-07-10 11:04:27,159 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 11:04:28,250 INFO [download_results] running in process 1464138
2025-07-10 11:04:28,251 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 11:04:28,252 INFO [download_results] publishing log messages to tcp://*:5562
2025-07-10 11:04:28,253 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-10 11:04:28,254 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/10jul25 to /results/SalishSea/nowcast-agrif.201702
2025-07-10 11:04:38,178 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete
2025-07-10 11:04:50,204 INFO [download_results] nowcast-agrif 2025-07-10 results files from orcinus-nowcast-agrif downloaded
2025-07-10 11:04:50,212 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-10 11:04:50,549 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 11:04:53,067 INFO [make_plots] running in process 1464146
2025-07-10 11:04:53,068 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 11:04:53,069 INFO [make_plots] publishing log messages to tcp://*:5562
2025-07-10 11:04:53,070 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-10 11:04:53,409 INFO [make_plots] preparing render list for 2025-07-10 NEMO nowcast-agrif research figures
2025-07-10 11:04:53,410 INFO [make_plots] added baynes_sound_surface figure to 2025-07-10 NEMO nowcast-agrif research render list
2025-07-10 11:04:53,411 INFO [make_plots] starting to render nemo nowcast-agrif research 10jul25 figures
2025-07-10 11:05:05,039 INFO [make_plots] finished rendering nemo nowcast-agrif research 10jul25 figures
2025-07-10 11:05:05,042 INFO [make_plots] nemo research plots for 2025-07-10 nowcast-agrif completed
2025-07-10 11:05:05,045 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-10 11:07:18,368 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18565068 = 2025-07-10 22:31:20 UTC, 93.9% complete
2025-07-10 11:09:38,145 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 597, 92.1% complete
2025-07-10 11:12:18,274 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18565200 = 2025-07-10 23:59:20 UTC, 100.0% complete
2025-07-10 11:14:38,065 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-10 11:14:38,165 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-10 11:14:38,555 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 11:14:39,860 INFO [download_wwatch3_results] running in process 1464278
2025-07-10 11:14:39,861 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 11:14:39,862 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-07-10 11:14:39,863 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-10 11:17:18,283 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-07-10 11:17:18,291 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-10 11:17:18,799 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 11:17:20,343 INFO [download_results] running in process 1464304
2025-07-10 11:17:20,345 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 11:17:20,346 INFO [download_results] publishing log messages to tcp://*:5564
2025-07-10 11:17:20,347 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-10 11:17:20,349 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/10jul25 to /results2/SalishSea/nowcast-green.202111
2025-07-10 11:17:20,511 INFO [download_wwatch3_results] forecast 2025-07-10 results files from arbutus.cloud-nowcast downloaded
2025-07-10 11:17:20,515 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-10 11:17:20,844 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 11:17:22,115 INFO [update_forecast_datasets] running in process 1464310
2025-07-10 11:17:22,116 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 11:17:22,117 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-07-10 11:17:22,118 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-10 11:17:22,119 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/10jul25 run
2025-07-10 11:17:22,173 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/10jul25 run
2025-07-10 11:17:22,174 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/10jul25 run
2025-07-10 11:17:22,283 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/10jul25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-07-10 11:17:22,285 INFO [update_forecast_datasets] wwatch3 2025-07-10 forecast rolling forecast datasets updated
2025-07-10 11:17:22,288 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-10 11:17:22,612 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-07-10 11:17:23,742 INFO [ping_erddap] running in process 1464315
2025-07-10 11:17:23,743 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 11:17:23,745 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-07-10 11:17:23,746 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-10 11:17:23,748 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-07-10 11:17:23,749 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-10 11:17:24,019 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 11:17:27,499 INFO [make_plots] running in process 1464320
2025-07-10 11:17:27,500 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 11:17:27,501 INFO [make_plots] publishing log messages to tcp://*:5562
2025-07-10 11:17:27,502 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-10 11:17:27,503 INFO [make_plots] preparing render list for 2025-07-10 WaveWatch3 forecast publish figures
2025-07-10 11:17:27,504 INFO [make_plots] added Halibut Bank figure to 2025-07-10 WaveWatch3 forecast publish render list
2025-07-10 11:17:27,505 INFO [make_plots] added Sentry Shoal figure to 2025-07-10 WaveWatch3 forecast publish render list
2025-07-10 11:17:27,507 INFO [make_plots] starting to render wwatch3 forecast publish 10jul25 figures
2025-07-10 11:17:27,963 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-07-10 11:17:34,519 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-07-10 11:17:39,704 INFO [make_plots] finished rendering wwatch3 forecast publish 10jul25 figures
2025-07-10 11:17:39,705 INFO [make_plots] wwatch3 publish plots for 2025-07-10 forecast completed
2025-07-10 11:17:39,709 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-10 15:04:04,946 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250710/18/
2025-07-10 15:04:04,947 INFO [collect_weather] 2025-07-10 2.5km weather forecast 18 collection complete
2025-07-10 15:04:04,951 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-10 15:04:05,417 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-07-10 15:04:05,423 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-07-10 15:04:05,429 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-07-10 15:04:05,433 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-07-11'], host='localhost')
2025-07-10 15:04:05,810 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250710/18/
2025-07-10 15:04:05,811 INFO [crop_gribs] 2025-07-10 18 GRIBs cropping complete
2025-07-10 15:04:05,814 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-10 15:04:06,790 INFO [download_weather] running in process 1466898
2025-07-10 15:04:06,791 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:06,792 INFO [download_weather] publishing log messages to tcp://*:5562
2025-07-10 15:04:06,793 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:06,806 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250710
2025-07-10 15:04:06,807 INFO [download_weather] running in process 1466896
2025-07-10 15:04:06,828 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:06,849 INFO [download_weather] publishing log messages to tcp://*:5563
2025-07-10 15:04:06,870 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:06,886 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250710
2025-07-10 15:04:06,887 INFO [collect_weather] running in process 1466899
2025-07-10 15:04:06,888 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:06,890 INFO [collect_weather] publishing log messages to tcp://*:5565
2025-07-10 15:04:06,890 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:06,892 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-07-10 15:04:07,662 INFO [crop_gribs] running in process 1466900
2025-07-10 15:04:07,663 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:07,664 INFO [crop_gribs] publishing log messages to tcp://*:5566
2025-07-10 15:04:07,665 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:07,666 INFO [crop_gribs] cropping 2025-07-11 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-07-10 15:04:07,678 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250711/00/
2025-07-10 15:04:10,930 INFO [download_results] nowcast-green 2025-07-10 results files from arbutus.cloud-nowcast downloaded
2025-07-10 15:04:10,935 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-10 15:04:11,272 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 15:04:11,278 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-07-10 15:04:11,283 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 15:04:11,289 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 15:04:11,293 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-07-10'], host='localhost')
2025-07-10 15:04:12,454 INFO [ping_erddap] running in process 1466934
2025-07-10 15:04:12,456 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:12,457 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-10 15:04:12,458 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:12,472 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-07-10 15:04:12,473 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-10 15:04:13,814 INFO [make_averaged_dataset] running in process 1466936
2025-07-10 15:04:13,815 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:13,816 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-07-10 15:04:13,817 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:13,818 INFO [make_averaged_dataset] creating day-averaged dataset for 10-Jul-2025 physics
2025-07-10 15:04:13,825 INFO [make_averaged_dataset] running in process 1466937
2025-07-10 15:04:13,826 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:13,827 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567
2025-07-10 15:04:13,828 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:13,829 INFO [make_averaged_dataset] creating day-averaged dataset for 10-Jul-2025 biology
2025-07-10 15:04:13,829 INFO [make_averaged_dataset] running in process 1466935
2025-07-10 15:04:13,830 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:13,830 INFO [make_averaged_dataset] publishing log messages to tcp://*:5568
2025-07-10 15:04:13,831 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:13,832 INFO [make_averaged_dataset] creating day-averaged dataset for 10-Jul-2025 chemistry
2025-07-10 15:04:14,615 INFO [make_plots] running in process 1466932
2025-07-10 15:04:14,616 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 15:04:14,617 INFO [make_plots] publishing log messages to tcp://*:5569
2025-07-10 15:04:14,618 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-10 15:04:14,965 INFO [make_plots] preparing render list for 2025-07-10 NEMO nowcast-green research figures
2025-07-10 15:04:17,629 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:18,393 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:19,156 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:19,987 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:20,806 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:21,580 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:22,329 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:23,238 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:24,076 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:24,974 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:25,826 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:26,584 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:27,505 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:27,508 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:27,510 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:27,511 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-07-10 NEMO nowcast-green research render list
2025-07-10 15:04:27,511 INFO [make_plots] starting to render nemo nowcast-green research 10jul25 figures
2025-07-10 15:05:13,097 INFO [make_averaged_dataset] day-averaged dataset for 10-Jul-2025 physics created
2025-07-10 15:05:13,101 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-10 15:05:18,948 INFO [make_averaged_dataset] day-averaged dataset for 10-Jul-2025 chemistry created
2025-07-10 15:05:18,952 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-10 15:05:35,006 INFO [make_averaged_dataset] day-averaged dataset for 10-Jul-2025 biology created
2025-07-10 15:05:35,009 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-10 15:07:49,725 INFO [download_weather] 2025-07-10 1km weather forecast 00 downloads complete
2025-07-10 15:07:49,728 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-07-10 15:07:57,900 INFO [download_weather] 2025-07-10 1km weather forecast 12 downloads complete
2025-07-10 15:07:57,903 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-07-10 16:11:08,932 INFO [make_plots] finished rendering nemo nowcast-green research 10jul25 figures
2025-07-10 16:11:08,945 INFO [make_plots] nemo research plots for 2025-07-10 nowcast-green completed
2025-07-10 16:11:09,011 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-10 21:07:31,698 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250711/00/
2025-07-10 21:07:31,700 INFO [collect_weather] 2025-07-11 2.5km weather forecast 00 collection complete
2025-07-10 21:07:31,704 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-10 21:07:32,243 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-07-10 21:07:32,252 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-07-11'], host='localhost')
2025-07-10 21:07:32,583 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250711/00/
2025-07-10 21:07:32,583 INFO [crop_gribs] 2025-07-11 00 GRIBs cropping complete
2025-07-10 21:07:32,586 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-10 21:07:34,220 INFO [collect_weather] running in process 1474320
2025-07-10 21:07:34,222 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 21:07:34,223 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-07-10 21:07:34,224 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-10 21:07:34,244 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-07-10 21:07:34,944 INFO [crop_gribs] running in process 1474322
2025-07-10 21:07:34,945 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-10 21:07:34,946 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-07-10 21:07:34,947 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-10 21:07:34,948 INFO [crop_gribs] cropping 2025-07-11 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-07-10 21:07:34,962 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250711/06/
2025-07-11 03:03:07,034 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250711/06/
2025-07-11 03:03:07,035 INFO [collect_weather] 2025-07-11 2.5km weather forecast 06 collection complete
2025-07-11 03:03:07,038 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-11 03:03:07,543 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,551 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,556 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,562 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,567 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,571 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,576 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,581 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,588 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,592 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,597 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,601 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,606 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,612 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,621 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-07-11 03:03:07,628 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-07-11 03:03:07,639 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-07-11 03:03:07,660 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-07-10'], host='localhost')
2025-07-11 03:03:07,665 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-07-11 03:03:07,673 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-07-11 03:03:07,681 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-07-11 03:03:10,554 INFO [collect_weather] running in process 1480046
2025-07-11 03:03:10,555 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:10,561 INFO [collect_NeahBay_ssh] running in process 1480045
2025-07-11 03:03:10,565 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:10,568 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-07-11 03:03:10,571 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2025-07-11 03:03:10,572 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:10,573 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:10,578 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-07-11
2025-07-11 03:03:10,582 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-07-11 03:03:11,517 INFO [collect_river_data] running in process 1480031
2025-07-11 03:03:11,518 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,519 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-07-11 03:03:11,520 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,520 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-07-10
2025-07-11 03:03:11,550 INFO [collect_river_data] running in process 1480033
2025-07-11 03:03:11,551 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,551 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-07-11 03:03:11,552 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,553 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-07-10
2025-07-11 03:03:11,560 INFO [collect_river_data] running in process 1480034
2025-07-11 03:03:11,561 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,562 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-07-11 03:03:11,563 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,563 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-07-10
2025-07-11 03:03:11,585 INFO [collect_river_data] running in process 1480035
2025-07-11 03:03:11,587 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,588 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-07-11 03:03:11,589 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,591 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-07-10
2025-07-11 03:03:11,613 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-07-11 03:03:11,614 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-07-10 completed
2025-07-11 03:03:11,622 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-07-11 03:03:11,623 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-07-10 completed
2025-07-11 03:03:11,625 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-07-11 03:03:11,626 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-07-10 completed
2025-07-11 03:03:11,671 INFO [collect_river_data] running in process 1480040
2025-07-11 03:03:11,674 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,675 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-07-11 03:03:11,676 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,677 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-07-10
2025-07-11 03:03:11,678 INFO [collect_river_data] running in process 1480026
2025-07-11 03:03:11,679 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,680 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-07-11 03:03:11,681 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,682 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-07-10
2025-07-11 03:03:11,687 INFO [collect_river_data] running in process 1480039
2025-07-11 03:03:11,688 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,689 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-07-11 03:03:11,690 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,692 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-07-10
2025-07-11 03:03:11,709 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-07-11 03:03:11,710 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-07-10 completed
2025-07-11 03:03:11,713 INFO [collect_river_data] running in process 1480037
2025-07-11 03:03:11,714 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,715 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-07-11 03:03:11,716 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,717 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-07-10
2025-07-11 03:03:11,719 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-07-11 03:03:11,720 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-07-10 completed
2025-07-11 03:03:11,722 INFO [collect_river_data] running in process 1480029
2025-07-11 03:03:11,723 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,724 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-07-11 03:03:11,725 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,726 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-07-10
2025-07-11 03:03:11,727 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:11,728 INFO [collect_river_data] running in process 1480036
2025-07-11 03:03:11,729 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,730 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-07-11 03:03:11,731 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,731 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-07-10
2025-07-11 03:03:11,743 INFO [collect_river_data] running in process 1480032
2025-07-11 03:03:11,744 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,745 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-07-11 03:03:11,746 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,747 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-07-10
2025-07-11 03:03:11,748 INFO [collect_river_data] running in process 1480030
2025-07-11 03:03:11,749 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,757 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-07-11 03:03:11,758 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-07-11 03:03:11,758 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-07-10 completed
2025-07-11 03:03:11,759 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,760 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-07-10
2025-07-11 03:03:11,762 INFO [collect_river_data] running in process 1480028
2025-07-11 03:03:11,763 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,763 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-07-11 03:03:11,764 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,765 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-07-10
2025-07-11 03:03:11,767 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-07-11 03:03:11,768 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-07-10 completed
2025-07-11 03:03:11,769 INFO [collect_river_data] running in process 1480038
2025-07-11 03:03:11,770 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,771 INFO [collect_river_data] publishing log messages to tcp://*:5578
2025-07-11 03:03:11,772 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,772 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-07-10
2025-07-11 03:03:11,779 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-07-11 03:03:11,780 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-07-10 completed
2025-07-11 03:03:11,788 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-07-11 03:03:11,789 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-07-10 completed
2025-07-11 03:03:11,808 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-07-11 03:03:11,810 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-07-10 completed
2025-07-11 03:03:11,812 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-07-10 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-07-11 03:03:11,813 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-07-10 completed
2025-07-11 03:03:11,859 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-07-10 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-07-11 03:03:11,860 INFO [collect_river_data] ECCC Capilano river data collection for 2025-07-10 completed
2025-07-11 03:03:11,918 INFO [get_vfpa_hadcp] running in process 1480044
2025-07-11 03:03:11,919 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:11,921 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5579
2025-07-11 03:03:11,922 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:11,923 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-07-10
2025-07-11 03:03:12,118 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:12,120 INFO [crop_gribs] running in process 1480047
2025-07-11 03:03:12,122 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:12,123 INFO [crop_gribs] publishing log messages to tcp://*:5590
2025-07-11 03:03:12,124 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:12,125 INFO [crop_gribs] cropping 2025-07-11 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-07-11 03:03:12,136 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250711/12/
2025-07-11 03:03:12,348 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-07-10 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-07-11 03:03:12,349 INFO [collect_river_data] ECCC Fraser river data collection for 2025-07-10 completed
2025-07-11 03:03:12,353 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-07-10 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-07-11 03:03:12,354 INFO [collect_river_data] ECCC Englishman river data collection for 2025-07-10 completed
2025-07-11 03:03:12,554 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:12,790 INFO [get_onc_ferry] running in process 1480043
2025-07-11 03:03:12,791 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:12,792 INFO [get_onc_ferry] publishing log messages to tcp://*:5566
2025-07-11 03:03:12,793 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:12,793 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-07-10
2025-07-11 03:03:12,818 INFO [get_onc_ctd] running in process 1480042
2025-07-11 03:03:12,821 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:12,821 INFO [get_onc_ctd] publishing log messages to tcp://*:5591
2025-07-11 03:03:12,822 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:12,823 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-07-10
2025-07-11 03:03:12,830 INFO [get_onc_ctd] running in process 1480041
2025-07-11 03:03:12,831 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:12,831 INFO [get_onc_ctd] publishing log messages to tcp://*:5567
2025-07-11 03:03:12,832 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:12,833 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-07-10
2025-07-11 03:03:12,845 INFO [collect_NeahBay_ssh] 2025-07-11T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-07-11 03:03:12,908 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:13,234 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:13,564 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:13,884 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:14,202 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:14,342 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-07-10 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202507.nc
2025-07-11 03:03:14,343 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-07 netcdf file
2025-07-11 03:03:14,516 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:14,830 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:15,132 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:15,436 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:15,742 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:16,042 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-11 03:03:16,375 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-07-11 03:03:16,746 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-07-11'], host='localhost')
2025-07-11 03:03:16,752 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-07-11 03:03:17,213 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-07-11 03:03:18,529 INFO [ping_erddap] running in process 1480375
2025-07-11 03:03:18,530 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:18,531 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-11 03:03:18,532 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:18,534 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-07-11 03:03:18,536 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-11 03:03:20,247 INFO [make_ssh_files] running in process 1480362
2025-07-11 03:03:20,265 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:20,271 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-07-11 03:03:20,272 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:20,272 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-07-11 NOAA Neah Bay 00Z observation and forecast values
2025-07-11 03:03:22,723 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d06.nc
2025-07-11 03:03:22,979 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d07.nc
2025-07-11 03:03:23,245 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d08.nc
2025-07-11 03:03:23,434 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d09.nc
2025-07-11 03:03:23,696 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d10.nc
2025-07-11 03:03:23,995 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d11.nc
2025-07-11 03:03:24,209 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d12.nc
2025-07-11 03:03:24,393 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d13.nc
2025-07-11 03:03:24,641 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d14.nc
2025-07-11 03:03:24,838 INFO [make_ssh_files] sea surface height boundary file for 2025-07-11 forecast2 run created
2025-07-11 03:03:24,841 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-07-11 03:03:25,212 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-07-11 03:03:25,218 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-07-11 03:03:26,642 INFO [make_201702_runoff_file] running in process 1480402
2025-07-11 03:03:26,643 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:26,644 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-07-11 03:03:26,645 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:26,715 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-07-11 03:03:26,877 INFO [make_201702_runoff_file] 2025-07-11 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-07-11 03:03:26,879 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-07-11 03:03:27,359 INFO [make_runoff_file] running in process 1480400
2025-07-11 03:03:27,360 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:27,361 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2025-07-11 03:03:27,362 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:27,365 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-07-10
2025-07-11 03:03:28,491 ERROR [make_runoff_file] no 2025-07-10 discharge obs for SkagitMountVernon: patching
2025-07-11 03:03:29,284 ERROR [make_runoff_file] no 2025-07-10 discharge obs for SnohomishMonroe: patching
2025-07-11 03:03:29,519 ERROR [make_runoff_file] no 2025-07-10 discharge obs for NisquallyMcKenna: patching
2025-07-11 03:03:29,648 ERROR [make_runoff_file] no 2025-07-10 discharge obs for GreenwaterGreenwater: patching
2025-07-11 03:03:31,146 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-07-10: /results/forcing/rivers/R202108Dailies_y2025m07d10.nc
2025-07-11 03:03:31,147 INFO [make_runoff_file] 2025-07-10 runoff file creation completed
2025-07-11 03:03:31,149 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-07-11 03:03:33,078 INFO [get_onc_ctd] 2025-07-10 ONC SCVIP CTD T&S file created
2025-07-11 03:03:33,084 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-07-11 03:03:33,299 INFO [get_onc_ctd] 2025-07-10 ONC SEVIP CTD T&S file created
2025-07-11 03:03:33,460 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-07-11 03:03:33,466 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-07-11 03:03:33,830 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-07-11 03:03:33,862 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2025-07-10
2025-07-11 03:03:34,684 INFO [ping_erddap] running in process 1480444
2025-07-11 03:03:34,685 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:34,686 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-11 03:03:34,687 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:34,689 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-07-11 03:03:34,690 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-11 03:03:34,861 INFO [ping_erddap] running in process 1480446
2025-07-11 03:03:34,862 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:03:34,863 INFO [ping_erddap] publishing log messages to tcp://*:5565
2025-07-11 03:03:34,864 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:03:34,866 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-07-11 03:03:35,071 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-11 03:03:57,128 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2025-07-10
2025-07-11 03:04:20,022 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2025-07-10
2025-07-11 03:04:45,956 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2025-07-10
2025-07-11 03:04:45,997 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,001 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,003 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,004 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2025-07-10
2025-07-11 03:04:46,048 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,052 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,054 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,055 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2025-07-10
2025-07-11 03:04:46,101 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,103 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,104 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2025-07-10
2025-07-11 03:04:46,143 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,145 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,146 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2025-07-10
2025-07-11 03:04:46,182 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,184 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-10; substituting empty dataset
2025-07-11 03:04:46,367 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2025-07-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-11 03:04:46,382 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2025-07-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-11 03:04:46,398 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2025-07-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-11 03:04:46,413 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2025-07-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-11 03:04:46,429 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2025-07-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-11 03:04:46,445 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2025-07-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-11 03:04:46,461 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-10 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-11 03:05:50,083 INFO [get_onc_ferry] 2025-07-10 ONC TWDP ferry data file created
2025-07-11 03:05:50,086 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2025-07-11 03:05:50,459 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2025-07-11 03:05:51,582 INFO [ping_erddap] running in process 1480475
2025-07-11 03:05:51,584 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 03:05:51,585 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-11 03:05:51,586 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-11 03:05:51,587 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2025-07-11 03:05:51,589 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-11 05:07:36,353 INFO [crop_gribs] crop_gribs 06 has watched for 8h; retrying remaining 1 file(s)
2025-07-11 05:07:38,250 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250711/06/
2025-07-11 05:07:38,251 INFO [crop_gribs] 2025-07-11 06 GRIBs cropping complete
2025-07-11 05:07:38,253 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-11 05:07:38,926 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-07-11 05:07:41,084 INFO [grib_to_netcdf] running in process 1481852
2025-07-11 05:07:41,085 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:07:41,086 INFO [grib_to_netcdf] publishing log messages to tcp://*:5563
2025-07-11 05:07:41,087 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:07:41,089 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-07-11 forecast2 run
2025-07-11 05:07:49,725 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d12.nc
2025-07-11 05:07:52,131 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d13.nc
2025-07-11 05:07:52,137 INFO [grib_to_netcdf] 2025-07-11 NEMO-atmos forcing file for forecast2 created
2025-07-11 05:07:52,140 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-07-11 05:07:52,494 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-07-11 05:07:52,500 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-07-11 05:07:52,505 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-07-11 05:07:52,510 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-07-11 05:07:53,774 INFO [upload_forcing] running in process 1481885
2025-07-11 05:07:53,775 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:07:53,776 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-07-11 05:07:53,777 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:07:53,779 INFO [upload_forcing] running in process 1481886
2025-07-11 05:07:53,780 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:07:53,782 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-11 05:07:53,783 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:07:53,822 INFO [upload_forcing] running in process 1481883
2025-07-11 05:07:53,823 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:07:53,824 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-07-11 05:07:53,825 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:07:53,841 INFO [upload_forcing] running in process 1481887
2025-07-11 05:07:53,842 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:07:53,843 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-11 05:07:53,844 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:07:55,309 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d10.nc
2025-07-11 05:07:55,310 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d10.nc
2025-07-11 05:07:55,389 INFO [upload_forcing] forecast2 2025-07-11 forcing files upload to optimum-hindcast completed
2025-07-11 05:07:55,393 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-11 05:07:55,394 INFO [upload_forcing] forecast2 2025-07-11 forcing files upload to orcinus-nowcast-agrif completed
2025-07-11 05:07:55,758 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-11 05:07:58,559 INFO [upload_forcing] forecast2 2025-07-11 forcing files upload to arbutus.cloud-nowcast completed
2025-07-11 05:07:58,563 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-11 05:07:58,933 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-11'], host='localhost')
2025-07-11 05:08:00,225 INFO [make_forcing_links] running in process 1481917
2025-07-11 05:08:00,228 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:08:00,229 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-07-11 05:08:00,230 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:08:01,437 INFO [make_forcing_links] forecast2 2025-07-11 forcing file links on arbutus.cloud-nowcast created
2025-07-11 05:08:01,441 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-11 05:08:01,805 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-11'], host='arbutus.cloud-nowcast')
2025-07-11 05:08:02,794 INFO [upload_forcing] forecast2 2025-07-11 forcing files upload to robot.graham completed
2025-07-11 05:08:02,799 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-11 05:08:04,152 INFO [run_NEMO] running in process 27710
2025-07-11 05:08:04,153 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:08:04,154 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-11 05:08:05,129 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:08:05,916 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-11 05:08:06,359 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/10jul25forecast2_2025-07-11T050805.201219-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-11 05:08:06,371 INFO [run_NEMO] forecast2 NEMO run for 2025-07-11 on arbutus.cloud-nowcast started
2025-07-11 05:08:06,375 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-11 05:08:06,707 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-07-11 05:08:08,665 INFO [watch_NEMO] running in process 27983
2025-07-11 05:08:08,667 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:08:08,668 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-11 05:08:08,671 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:08:08,736 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-11 05:13:08,737 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18567805 = 2025-07-12 04:56:00 UTC, 16.5% complete
2025-07-11 05:18:08,739 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18568274 = 2025-07-12 10:08:40 UTC, 33.8% complete
2025-07-11 05:23:08,740 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18568732 = 2025-07-12 15:14:00 UTC, 50.8% complete
2025-07-11 05:28:08,740 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18569200 = 2025-07-12 20:26:00 UTC, 68.1% complete
2025-07-11 05:33:08,741 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18569675 = 2025-07-13 01:42:40 UTC, 85.7% complete
2025-07-11 05:38:08,742 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18570060 = 2025-07-13 05:59:20 UTC, 100.0% complete
2025-07-11 05:43:08,746 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-07-11 05:43:08,750 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-11 05:43:09,234 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-11'], host='arbutus.cloud-nowcast')
2025-07-11 05:43:09,242 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-11'], host='arbutus.cloud-nowcast')
2025-07-11 05:43:09,246 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-10'], host='localhost')
2025-07-11 05:43:10,600 INFO [download_results] running in process 1482328
2025-07-11 05:43:10,601 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:43:10,602 INFO [download_results] publishing log messages to tcp://*:5563
2025-07-11 05:43:10,604 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:43:10,605 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/10jul25 to /results/SalishSea/forecast2.202111
2025-07-11 05:43:12,754 INFO [make_ww3_wind_file] running in process 28389
2025-07-11 05:43:12,755 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:43:12,757 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-07-11 05:43:12,759 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:43:12,760 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-07-11 forecast2 run
2025-07-11 05:43:13,221 INFO [make_ww3_current_file] running in process 28388
2025-07-11 05:43:13,223 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:43:13,224 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-07-11 05:43:13,225 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:43:13,226 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-07-11 forecast2 run
2025-07-11 05:43:21,679 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-07-11 forecast2 run
2025-07-11 05:43:21,681 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-07-11 05:44:12,064 INFO [download_results] forecast2 2025-07-10 results files from arbutus.cloud-nowcast downloaded
2025-07-11 05:44:12,069 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-11 05:44:12,441 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-07-10'], host='localhost')
2025-07-11 05:44:14,430 INFO [make_CHS_currents_file] running in process 1482339
2025-07-11 05:44:14,432 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:44:14,433 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-07-11 05:44:14,434 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:44:49,508 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-10 for forecast2
2025-07-11 05:44:49,512 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-11 05:44:49,884 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-07-10'], host='localhost')
2025-07-11 05:44:51,024 INFO [update_forecast_datasets] running in process 1482363
2025-07-11 05:44:51,025 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:44:51,027 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-07-11 05:44:51,028 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:44:51,029 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/10jul25 run
2025-07-11 05:46:48,037 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-07-11 forecast2 run
2025-07-11 05:46:48,054 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-07-11 05:46:48,388 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-11'], host='arbutus.cloud-nowcast')
2025-07-11 05:46:50,500 INFO [run_ww3] running in process 28946
2025-07-11 05:46:50,501 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:46:50,502 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-07-11 05:46:50,503 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:46:50,507 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-07-11T054650.497787-0700
2025-07-11 05:46:50,508 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-07-11T054650.497787-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-07-11 05:46:50,518 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-07-11 on arbutus.cloud-nowcast started
2025-07-11 05:46:50,546 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-11 05:46:50,918 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-07-11 05:46:52,908 INFO [watch_ww3] running in process 29033
2025-07-11 05:46:52,909 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:46:52,910 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-07-11 05:46:52,911 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:46:52,943 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-11 05:49:06,191 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/10jul25 run: /results/SalishSea/rolling-forecasts/nemo
2025-07-11 05:49:06,192 INFO [update_forecast_datasets] nemo 2025-07-10 forecast2 rolling forecast datasets updated
2025-07-11 05:49:06,195 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-11 05:49:06,569 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-07-11 05:49:06,575 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-07-10'], host='localhost')
2025-07-11 05:49:06,581 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-07-10'], host='localhost')
2025-07-11 05:49:07,883 INFO [ping_erddap] running in process 1482469
2025-07-11 05:49:07,883 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:49:07,885 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-11 05:49:07,886 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:49:08,056 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-07-11 05:49:08,058 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-11 05:49:09,135 INFO [make_surface_current_tiles] running in process 1482472
2025-07-11 05:49:09,136 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:49:09,137 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-07-11 05:49:09,138 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:49:09,139 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-07-10 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/10jul25
2025-07-11 05:49:09,937 INFO [make_plots] running in process 1482471
2025-07-11 05:49:09,939 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 05:49:09,940 INFO [make_plots] publishing log messages to tcp://*:5564
2025-07-11 05:49:09,946 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-11 05:49:10,441 INFO [make_plots] preparing render list for 2025-07-10 NEMO forecast2 publish figures
2025-07-11 05:49:10,584 INFO [make_plots] added Website_thumbnail figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,585 INFO [make_plots] added Threshold_website figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,588 INFO [make_plots] added PA_tidal_predictions figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,590 INFO [make_plots] added Boundary Bay figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,591 INFO [make_plots] added Campbell River figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,592 INFO [make_plots] added Cherry Point figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,593 INFO [make_plots] added Friday Harbor figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,595 INFO [make_plots] added Halfmoon Bay figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,595 INFO [make_plots] added Nanaimo figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,596 INFO [make_plots] added Neah Bay figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,597 INFO [make_plots] added New Westminster figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,598 INFO [make_plots] added Patricia Bay figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,598 INFO [make_plots] added Point Atkinson figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,599 INFO [make_plots] added Port Renfrew figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,599 INFO [make_plots] added Sand Heads figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,600 INFO [make_plots] added Sandy Cove figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,600 INFO [make_plots] added Squamish figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,601 INFO [make_plots] added Victoria figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,601 INFO [make_plots] added Woodwards Landing figure to 2025-07-10 NEMO forecast2 publish render list
2025-07-11 05:49:10,602 INFO [make_plots] starting to render nemo forecast2 publish 10jul25 figures
2025-07-11 05:51:52,943 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 93, 17.2% complete
2025-07-11 05:56:52,944 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 201, 37.2% complete
2025-07-11 05:57:20,023 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-07-11 05:57:20,024 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-07-11 05:57:44,458 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 05:59:00,608 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-07-11 05:59:00,609 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-07-11 05:59:25,645 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:00:07,982 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:00:22,941 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/10jul25
2025-07-11 06:00:26,654 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:00:44,291 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:01:02,319 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:01:18,065 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:01:35,648 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:01:52,945 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 309, 57.2% complete
2025-07-11 06:01:53,235 WARNING [make_plots] invalid station number for Squamish station: None
2025-07-11 06:01:53,236 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-07-11 06:02:10,095 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:02:26,143 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-07-06 00:00:00Z to 2025-07-13 05:55:00Z
2025-07-11 06:02:43,288 INFO [make_plots] finished rendering nemo forecast2 publish 10jul25 figures
2025-07-11 06:02:43,291 INFO [make_plots] nemo publish plots for 2025-07-10 forecast2 completed
2025-07-11 06:02:43,297 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-11 06:02:43,656 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-07-10'], host='localhost')
2025-07-11 06:02:46,906 INFO [make_feeds] running in process 1484521
2025-07-11 06:02:46,907 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 06:02:46,909 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-07-11 06:02:46,910 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-07-11 06:03:02,011 INFO [make_feeds] ATOM feeds for 2025-07-10 forecast2 run completed
2025-07-11 06:03:02,014 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-07-11 06:03:02,325 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-07-11 06:03:03,425 INFO [clear_checklist] running in process 1484543
2025-07-11 06:03:03,426 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 06:03:03,427 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-07-11 06:03:03,428 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-07-11 06:03:03,429 INFO [clear_checklist] requesting that manager clear system state checklist
2025-07-11 06:03:03,432 INFO [manager] writing checklist to log file
2025-07-11 06:03:03,478 INFO [manager] checklist cleared
2025-07-11 06:03:03,480 INFO [clear_checklist] nowcast system checklist cleared
2025-07-11 06:03:03,482 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-07-11 06:03:04,559 INFO [rotate_logs] running in process 1484549
2025-07-11 06:03:04,560 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-11 06:03:04,561 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-07-11 06:03:04,562 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-07-11 06:03:04,563 INFO [rotate_logs] rotating log files