2025-07-06 03:57:23,518 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-07-06 03:57:23,523 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-07-06 03:57:23,526 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-07-06 03:57:23,527 INFO [rotate_logs] log files rotated
2025-07-06 03:57:23,531 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-07-06 03:59:26,730 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/05jul25
2025-07-06 03:59:26,752 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-07-05 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/05jul25
2025-07-06 03:59:26,753 INFO [make_surface_current_tiles] surface current tile figures for 2025-07-05 forecast2 completed
2025-07-06 03:59:27,058 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-07-06 04:00:18,283 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 309, 57.2% complete
2025-07-06 04:05:18,286 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 387, 71.7% complete
2025-07-06 04:10:18,301 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 462, 85.6% complete
2025-07-06 04:15:18,289 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-06 04:20:18,285 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-06 04:20:18,288 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-06 04:20:18,695 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 04:20:19,993 INFO [download_wwatch3_results] running in process 1368871
2025-07-06 04:20:19,994 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 04:20:19,996 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-07-06 04:20:19,997 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-06 04:21:07,457 INFO [download_wwatch3_results] forecast2 2025-07-06 results files from arbutus.cloud-nowcast downloaded
2025-07-06 04:21:07,461 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-06 04:21:07,620 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 04:21:08,892 INFO [update_forecast_datasets] running in process 1368882
2025-07-06 04:21:08,894 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 04:21:08,895 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-07-06 04:21:08,896 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-06 04:21:08,897 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/06jul25 run
2025-07-06 04:21:08,901 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/06jul25 run
2025-07-06 04:21:08,902 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/06jul25 run
2025-07-06 04:21:34,123 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/06jul25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-07-06 04:21:34,124 INFO [update_forecast_datasets] wwatch3 2025-07-06 forecast2 rolling forecast datasets updated
2025-07-06 04:21:34,128 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-06 04:21:34,289 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-07-06 04:21:35,638 INFO [ping_erddap] running in process 1368898
2025-07-06 04:21:35,640 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 04:21:35,641 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-06 04:21:35,642 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-06 04:21:35,644 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-07-06 04:21:35,646 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-06 04:21:35,789 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 04:21:39,330 INFO [make_plots] running in process 1368903
2025-07-06 04:21:39,331 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 04:21:39,332 INFO [make_plots] publishing log messages to tcp://*:5563
2025-07-06 04:21:39,333 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-06 04:21:39,334 INFO [make_plots] preparing render list for 2025-07-06 WaveWatch3 forecast2 publish figures
2025-07-06 04:21:39,335 INFO [make_plots] added Halibut Bank figure to 2025-07-06 WaveWatch3 forecast2 publish render list
2025-07-06 04:21:39,336 INFO [make_plots] added Sentry Shoal figure to 2025-07-06 WaveWatch3 forecast2 publish render list
2025-07-06 04:21:39,336 INFO [make_plots] starting to render wwatch3 forecast2 publish 06jul25 figures
2025-07-06 04:21:39,771 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-07-06 04:21:44,709 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-07-06 04:21:49,170 INFO [make_plots] finished rendering wwatch3 forecast2 publish 06jul25 figures
2025-07-06 04:21:49,171 INFO [make_plots] wwatch3 publish plots for 2025-07-06 forecast2 completed
2025-07-06 04:21:49,174 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-06 09:09:28,054 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250706/12/
2025-07-06 09:09:28,055 INFO [collect_weather] 2025-07-06 2.5km weather forecast 12 collection complete
2025-07-06 09:09:28,058 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-06 09:09:28,415 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-07-05'], host='localhost')
2025-07-06 09:09:28,421 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-07-05'], host='localhost')
2025-07-06 09:09:28,426 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-07-05'], host='localhost')
2025-07-06 09:09:28,432 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-07-05'], host='localhost')
2025-07-06 09:09:28,436 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-07-06 09:09:28,445 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-07-06 09:09:28,450 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-07-06 09:09:28,459 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-07-06 09:09:28,463 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-07-06 09:09:28,527 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250706/12/
2025-07-06 09:09:28,528 INFO [crop_gribs] 2025-07-06 12 GRIBs cropping complete
2025-07-06 09:09:28,530 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-06 09:09:29,115 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-07-06 09:09:30,079 INFO [collect_weather] running in process 1372140
2025-07-06 09:09:30,080 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:30,081 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-07-06 09:09:30,082 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:30,086 INFO [collect_NeahBay_ssh] running in process 1372138
2025-07-06 09:09:30,087 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:30,087 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-07-06 09:09:30,088 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:30,089 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-07-06
2025-07-06 09:09:30,102 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-07-06 09:09:30,284 INFO [download_live_ocean] running in process 1372139
2025-07-06 09:09:30,284 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:30,285 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-07-06 09:09:30,286 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:30,288 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-07-06
2025-07-06 09:09:30,634 INFO [collect_river_data] running in process 1372136
2025-07-06 09:09:30,635 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:30,636 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-07-06 09:09:30,637 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:30,637 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-07-05
2025-07-06 09:09:30,668 INFO [collect_river_data] running in process 1372134
2025-07-06 09:09:30,670 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:30,671 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-07-06 09:09:30,672 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:30,673 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-07-05
2025-07-06 09:09:30,687 INFO [collect_river_data] running in process 1372135
2025-07-06 09:09:30,688 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:30,688 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-07-06 09:09:30,689 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:30,690 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-07-05
2025-07-06 09:09:30,747 INFO [collect_river_data] running in process 1372132
2025-07-06 09:09:30,748 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:30,749 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-07-06 09:09:30,749 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:30,750 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-07-05
2025-07-06 09:09:30,795 INFO [make_turbidity_file] running in process 1372137
2025-07-06 09:09:30,797 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:30,797 INFO [make_turbidity_file] publishing log messages to tcp://*:5569
2025-07-06 09:09:30,798 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:30,799 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-07-06
2025-07-06 09:09:31,038 INFO [crop_gribs] running in process 1372141
2025-07-06 09:09:31,039 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:31,041 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-07-06 09:09:31,042 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:31,043 INFO [crop_gribs] cropping 2025-07-06 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-07-06 09:09:31,046 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-07-05&endDT=2025-07-05 "HTTP/1.1 200 "
2025-07-06 09:09:31,053 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250706/18/
2025-07-06 09:09:31,079 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-07-05&endDT=2025-07-05 "HTTP/1.1 200 "
2025-07-06 09:09:31,084 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-07-05 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-07-06 09:09:31,085 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-07-05 completed
2025-07-06 09:09:31,087 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-06 09:09:31,152 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-07-05 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-07-06 09:09:31,153 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-07-05 completed
2025-07-06 09:09:31,155 INFO [grib_to_netcdf] running in process 1372212
2025-07-06 09:09:31,156 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:31,157 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2025-07-06 09:09:31,158 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:31,159 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-07-06 nowcast and forecast runs
2025-07-06 09:09:31,207 INFO [download_live_ocean] downloaded 10214580 bytes from https://liveocean.apl.uw.edu/output/f2025.07.06/ubc.nc to /results/forcing/LiveOcean/downloaded/20250706/low_passed_UBC.nc
2025-07-06 09:09:31,208 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-07-06 09:09:31,210 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-07-05&endDT=2025-07-05 "HTTP/1.1 200 "
2025-07-06 09:09:31,211 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-07-05&endDT=2025-07-05 "HTTP/1.1 200 "
2025-07-06 09:09:31,234 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-06 09:09:31,244 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-07-05 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-07-06 09:09:31,244 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-07-05 completed
2025-07-06 09:09:31,248 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-07-05 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-07-06 09:09:31,249 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-07-05 completed
2025-07-06 09:09:31,349 INFO [collect_NeahBay_ssh] 2025-07-06T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-07-06 09:09:31,405 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-06 09:09:31,551 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-06 09:09:31,728 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-07-06 09:09:31,877 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 09:09:32,246 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250706/low_passed_UBC.nc
2025-07-06 09:09:32,247 INFO [download_live_ocean] 2025-07-06 Live Ocean file for Salish Sea western boundary downloaded
2025-07-06 09:09:32,250 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-07-06 09:09:32,599 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-07-06'], host='localhost')
2025-07-06 09:09:33,176 INFO [make_turbidity_file] 2025-07-06 Fraser River turbidity file creation complete
2025-07-06 09:09:33,178 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-07-06 09:09:34,602 INFO [make_ssh_files] running in process 1372273
2025-07-06 09:09:34,603 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:34,604 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2025-07-06 09:09:34,606 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:34,607 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-07-06 NOAA Neah Bay 06Z observation and forecast values
2025-07-06 09:09:34,650 INFO [make_live_ocean_files] running in process 1372286
2025-07-06 09:09:34,651 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:34,652 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2025-07-06 09:09:34,653 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:34,654 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-07-06 Live Ocean run
2025-07-06 09:09:36,293 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d02.nc
2025-07-06 09:09:36,464 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d03.nc
2025-07-06 09:09:36,635 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d04.nc
2025-07-06 09:09:36,808 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d05.nc
2025-07-06 09:09:37,017 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d06.nc
2025-07-06 09:09:37,207 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d07.nc
2025-07-06 09:09:37,386 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d08.nc
2025-07-06 09:09:37,576 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d09.nc
2025-07-06 09:09:37,690 INFO [make_ssh_files] sea surface height boundary file for 2025-07-06 nowcast run created
2025-07-06 09:09:37,693 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-07-06 09:09:37,832 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-07-06 09:09:37,837 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-07-06 09:09:39,150 INFO [make_201702_runoff_file] running in process 1372319
2025-07-06 09:09:39,151 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:39,152 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2025-07-06 09:09:39,154 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:39,180 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-07-06 09:09:39,265 INFO [make_201702_runoff_file] 2025-07-06 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-07-06 09:09:39,267 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-07-06 09:09:39,795 INFO [make_runoff_file] running in process 1372317
2025-07-06 09:09:39,796 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:09:39,797 INFO [make_runoff_file] publishing log messages to tcp://*:5563
2025-07-06 09:09:39,798 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:09:39,801 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-07-05
2025-07-06 09:09:41,794 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m07d06.nc
2025-07-06 09:09:42,584 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-07-05: /results/forcing/rivers/R202108Dailies_y2025m07d05.nc
2025-07-06 09:09:42,585 INFO [make_runoff_file] 2025-07-05 runoff file creation completed
2025-07-06 09:09:42,587 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-07-06 09:09:49,167 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d07.nc
2025-07-06 09:09:53,156 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d08.nc
2025-07-06 09:09:53,170 INFO [grib_to_netcdf] 2025-07-06 NEMO-atmos forcing file for nowcast+ created
2025-07-06 09:09:53,173 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-07-06 09:10:16,721 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d06.nc
2025-07-06 09:10:16,721 INFO [make_live_ocean_files] 2025-07-06 Live Ocean western boundary conditions files created
2025-07-06 09:10:16,723 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-07-06 09:10:16,886 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-07-06 09:10:16,891 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-07-06 09:10:16,896 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-07-06 09:10:16,900 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-07-06 09:10:16,905 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-07-06 09:10:17,998 INFO [ping_erddap] running in process 1372366
2025-07-06 09:10:17,999 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:10:18,000 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-06 09:10:18,001 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:10:18,003 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-07-06 09:10:18,005 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-06 09:10:18,083 INFO [upload_forcing] running in process 1372371
2025-07-06 09:10:18,084 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:10:18,085 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-06 09:10:18,086 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:10:18,087 INFO [upload_forcing] running in process 1372369
2025-07-06 09:10:18,088 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:10:18,089 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-07-06 09:10:18,090 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:10:18,129 INFO [upload_forcing] running in process 1372370
2025-07-06 09:10:18,130 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:10:18,131 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-06 09:10:18,132 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:10:18,135 INFO [upload_forcing] running in process 1372368
2025-07-06 09:10:18,136 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:10:18,137 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-07-06 09:10:18,138 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:10:20,206 INFO [upload_forcing] nowcast+ 2025-07-06 forcing files upload to optimum-hindcast completed
2025-07-06 09:10:20,210 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-06 09:10:20,268 INFO [upload_forcing] nowcast+ 2025-07-06 forcing files upload to orcinus-nowcast-agrif completed
2025-07-06 09:10:20,383 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-06 09:10:28,673 INFO [upload_forcing] nowcast+ 2025-07-06 forcing files upload to robot.graham completed
2025-07-06 09:10:28,677 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-06 09:10:32,648 INFO [upload_forcing] nowcast+ 2025-07-06 forcing files upload to arbutus.cloud-nowcast completed
2025-07-06 09:10:32,653 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-06 09:10:32,809 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 09:10:34,435 INFO [make_forcing_links] running in process 1372411
2025-07-06 09:10:34,436 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:10:34,437 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-07-06 09:10:34,439 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:10:36,080 INFO [make_forcing_links] nowcast+ 2025-07-06 forcing file links on arbutus.cloud-nowcast created
2025-07-06 09:10:36,084 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-06 09:10:36,240 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-06'], host='arbutus.cloud-nowcast')
2025-07-06 09:10:38,248 INFO [run_NEMO] running in process 12508
2025-07-06 09:10:38,249 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:10:38,251 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-06 09:10:38,358 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:10:39,006 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-06 09:10:39,311 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/06jul25nowcast_2025-07-06T091038.276068-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-06 09:10:39,313 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-06 09:10:39,322 INFO [run_NEMO] nowcast NEMO run for 2025-07-06 on arbutus.cloud-nowcast started
2025-07-06 09:10:39,471 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-07-06 09:10:41,617 INFO [watch_NEMO] running in process 12781
2025-07-06 09:10:41,618 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:10:41,619 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-06 09:10:41,620 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:10:41,761 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-06 09:15:41,762 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18554875 = 2025-07-06 05:16:00 UTC, 22.0% complete
2025-07-06 09:20:41,761 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18555404 = 2025-07-06 11:08:40 UTC, 46.5% complete
2025-07-06 09:25:41,754 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18555910 = 2025-07-06 16:46:00 UTC, 69.9% complete
2025-07-06 09:30:41,751 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18556411 = 2025-07-06 22:20:00 UTC, 93.1% complete
2025-07-06 09:35:41,757 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-07-06 09:35:41,760 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-06 09:35:42,106 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 09:35:42,114 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 09:35:43,355 INFO [download_results] running in process 1372725
2025-07-06 09:35:43,356 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:35:43,358 INFO [download_results] publishing log messages to tcp://*:5563
2025-07-06 09:35:43,359 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:35:43,360 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/06jul25 to /results/SalishSea/nowcast-blue.202111
2025-07-06 09:35:43,398 INFO [make_forcing_links] running in process 1372724
2025-07-06 09:35:43,399 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:35:43,400 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-07-06 09:35:43,401 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:35:44,364 INFO [make_forcing_links] ssh 2025-07-06 forcing file links on arbutus.cloud-nowcast created
2025-07-06 09:35:44,368 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-06 09:35:44,527 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-06'], host='arbutus.cloud-nowcast')
2025-07-06 09:35:46,677 INFO [run_NEMO] running in process 13197
2025-07-06 09:35:46,678 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:35:46,679 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-07-06 09:35:46,680 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:35:47,469 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-06 09:35:47,787 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/06jul25forecast_2025-07-06T093546.775271-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-06 09:35:47,798 INFO [run_NEMO] forecast NEMO run for 2025-07-06 on arbutus.cloud-nowcast started
2025-07-06 09:35:47,816 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-06 09:35:47,979 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-07-06 09:35:50,566 INFO [watch_NEMO] running in process 13473
2025-07-06 09:35:50,568 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:35:50,569 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-06 09:35:50,571 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:35:50,662 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-06 09:36:27,817 INFO [download_results] nowcast 2025-07-06 results files from arbutus.cloud-nowcast downloaded
2025-07-06 09:36:27,822 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-06 09:36:27,983 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 09:36:27,988 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-07-05'], host='localhost')
2025-07-06 09:36:27,993 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 09:36:30,020 INFO [make_CHS_currents_file] running in process 1372750
2025-07-06 09:36:30,021 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:36:30,022 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-07-06 09:36:30,023 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:36:31,087 INFO [make_plots] running in process 1372747
2025-07-06 09:36:31,089 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:36:31,090 INFO [make_plots] publishing log messages to tcp://*:5564
2025-07-06 09:36:31,092 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:36:31,118 INFO [make_plots] running in process 1372749
2025-07-06 09:36:31,158 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 09:36:31,165 INFO [make_plots] publishing log messages to tcp://*:5565
2025-07-06 09:36:31,166 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-06 09:36:31,551 INFO [make_plots] preparing render list for 2025-07-05 NEMO nowcast-blue comparison figures
2025-07-06 09:36:31,561 INFO [make_plots] preparing render list for 2025-07-06 NEMO nowcast-blue research figures
2025-07-06 09:36:31,684 INFO [make_plots] added SH_wind figure to 2025-07-05 NEMO nowcast-blue comparison render list
2025-07-06 09:36:31,686 INFO [make_plots] added Compare_VENUS_East figure to 2025-07-05 NEMO nowcast-blue comparison render list
2025-07-06 09:36:31,687 INFO [make_plots] added Compare_VENUS_Central figure to 2025-07-05 NEMO nowcast-blue comparison render list
2025-07-06 09:36:31,688 INFO [make_plots] starting to render nemo nowcast comparison 05jul25 figures
2025-07-06 09:36:33,146 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-blue research render list
2025-07-06 09:36:34,444 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-blue research render list
2025-07-06 09:36:34,445 INFO [make_plots] added Currents_sections_and_surface figure to 2025-07-06 NEMO nowcast-blue research render list
2025-07-06 09:36:34,447 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-07-06 NEMO nowcast-blue research render list
2025-07-06 09:36:34,448 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-07-06 NEMO nowcast-blue research render list
2025-07-06 09:36:34,449 INFO [make_plots] starting to render nemo nowcast research 06jul25 figures
2025-07-06 09:37:05,341 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-06 for nowcast
2025-07-06 09:37:05,344 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-06 09:37:45,253 INFO [make_plots] finished rendering nemo nowcast research 06jul25 figures
2025-07-06 09:37:45,256 INFO [make_plots] nemo research plots for 2025-07-06 nowcast completed
2025-07-06 09:37:45,272 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-06 09:38:00,107 INFO [make_plots] finished rendering nemo nowcast comparison 05jul25 figures
2025-07-06 09:38:00,110 INFO [make_plots] nemo comparison plots for 2025-07-05 nowcast completed
2025-07-06 09:38:00,114 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-06 09:40:50,660 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18557024 = 2025-07-07 05:08:40 UTC, 14.3% complete
2025-07-06 09:45:50,661 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18557526 = 2025-07-07 10:43:20 UTC, 29.8% complete
2025-07-06 09:50:50,659 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18558026 = 2025-07-07 16:16:40 UTC, 45.2% complete
2025-07-06 09:55:50,658 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18558530 = 2025-07-07 21:52:40 UTC, 60.8% complete
2025-07-06 10:00:50,657 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18559007 = 2025-07-08 03:10:40 UTC, 75.5% complete
2025-07-06 10:05:50,658 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18559495 = 2025-07-08 08:36:00 UTC, 90.6% complete
2025-07-06 10:10:50,662 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-07-06 10:10:50,669 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-06 10:10:51,091 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-06'], host='arbutus.cloud-nowcast')
2025-07-06 10:10:51,100 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-06'], host='arbutus.cloud-nowcast')
2025-07-06 10:10:51,105 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:10:51,111 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:10:51,117 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:10:51,124 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:10:51,128 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:10:52,353 INFO [upload_forcing] running in process 1374082
2025-07-06 10:10:52,354 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:52,356 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-07-06 10:10:52,357 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:52,358 INFO [upload_forcing] running in process 1374083
2025-07-06 10:10:52,359 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:52,360 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-06 10:10:52,361 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:52,362 INFO [download_results] running in process 1374084
2025-07-06 10:10:52,364 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:52,364 INFO [download_results] publishing log messages to tcp://*:5565
2025-07-06 10:10:52,365 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:52,366 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/06jul25 to /results/SalishSea/forecast.202111
2025-07-06 10:10:52,368 INFO [upload_forcing] running in process 1374080
2025-07-06 10:10:52,369 INFO [upload_forcing] running in process 1374081
2025-07-06 10:10:52,369 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:52,370 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:52,371 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-06 10:10:52,372 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-07-06 10:10:52,373 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:52,374 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:53,027 INFO [upload_forcing] turbidity 2025-07-06 forcing files upload to optimum-hindcast completed
2025-07-06 10:10:53,030 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-06 10:10:53,083 INFO [upload_forcing] turbidity 2025-07-06 forcing files upload to orcinus-nowcast-agrif completed
2025-07-06 10:10:53,200 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-06 10:10:53,354 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:10:53,433 INFO [upload_forcing] turbidity 2025-07-06 forcing files upload to arbutus.cloud-nowcast completed
2025-07-06 10:10:53,437 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-06 10:10:53,594 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:10:53,897 INFO [make_ww3_wind_file] running in process 14074
2025-07-06 10:10:53,898 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:53,898 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-07-06 10:10:53,910 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:53,912 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-07-06 forecast run
2025-07-06 10:10:54,692 INFO [make_forcing_links] running in process 1374116
2025-07-06 10:10:54,693 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:54,694 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-07-06 10:10:54,696 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:55,010 INFO [make_forcing_links] running in process 1374118
2025-07-06 10:10:55,012 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:55,013 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2025-07-06 10:10:55,014 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:55,470 INFO [make_forcing_links] nowcast-agrif 2025-07-06 forcing file links on orcinus-nowcast-agrif created
2025-07-06 10:10:55,475 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-06 10:10:55,640 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:10:56,528 INFO [make_forcing_links] nowcast-green 2025-07-06 forcing file links on arbutus.cloud-nowcast created
2025-07-06 10:10:56,532 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-06 10:10:56,700 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-06'], host='arbutus.cloud-nowcast')
2025-07-06 10:10:57,052 INFO [run_NEMO_agrif] running in process 1374132
2025-07-06 10:10:57,054 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:10:57,056 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2025-07-06 10:10:57,058 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:10:59,280 INFO [upload_forcing] turbidity 2025-07-06 forcing files upload to robot.graham completed
2025-07-06 10:10:59,284 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-06 10:11:02,655 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-06 10:11:03,641 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/06jul25nowcast-green_2025-07-06T101100.245846-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-06 10:11:03,709 INFO [run_NEMO] nowcast-green NEMO run for 2025-07-06 on arbutus.cloud-nowcast started
2025-07-06 10:11:03,712 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-06 10:11:03,912 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-07-06 10:11:05,896 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-07-06 forecast run
2025-07-06 10:11:05,941 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-07-06 10:11:06,110 INFO [watch_NEMO] running in process 14982
2025-07-06 10:11:06,114 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:11:06,115 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-06 10:11:06,116 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:11:06,199 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-06 10:11:13,769 INFO [run_NEMO_agrif] job id for 06jul25nowcast-agrif: 11435164.orca2.ibb
2025-07-06 10:11:13,771 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2025-07-06 queued on orcinus-nowcast-agrif
2025-07-06 10:11:13,776 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2025-07-06 10:11:13,940 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11435164.orca2.ibb'], host='localhost')
2025-07-06 10:11:15,436 INFO [watch_NEMO_agrif] running in process 1374151
2025-07-06 10:11:15,438 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:11:15,439 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5563
2025-07-06 10:11:15,441 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:11:16,260 INFO [watch_NEMO_agrif] watching 06jul25nowcast-agrif job 11435164 on orcinus-nowcast-agrif
2025-07-06 10:11:16,496 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif job 11435164 is queued on orcinus-nowcast-agrif
2025-07-06 10:12:17,548 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2025-07-06 10:12:30,137 INFO [download_results] forecast 2025-07-06 results files from arbutus.cloud-nowcast downloaded
2025-07-06 10:12:30,142 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-06 10:12:30,319 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:12:32,791 INFO [make_CHS_currents_file] running in process 1374188
2025-07-06 10:12:32,792 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:12:32,793 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2025-07-06 10:12:32,794 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:13:20,212 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-06 for forecast
2025-07-06 10:13:20,216 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-06 10:13:20,383 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:13:21,547 INFO [update_forecast_datasets] running in process 1374209
2025-07-06 10:13:21,548 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:13:21,549 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-07-06 10:13:21,550 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:13:21,551 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/06jul25 run
2025-07-06 10:13:21,909 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/06jul25 run: /results/SalishSea/rolling-forecasts/nemo
2025-07-06 10:13:21,910 INFO [update_forecast_datasets] nemo 2025-07-06 forecast rolling forecast datasets updated
2025-07-06 10:13:21,913 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-06 10:13:22,072 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-07-06 10:13:22,078 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:13:22,082 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:13:23,153 INFO [ping_erddap] running in process 1374214
2025-07-06 10:13:23,154 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:13:23,155 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-06 10:13:23,156 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:13:23,166 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-07-06 10:13:23,167 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-06 10:13:24,315 INFO [make_surface_current_tiles] running in process 1374217
2025-07-06 10:13:24,317 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:13:24,318 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2025-07-06 10:13:24,319 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:13:24,321 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-07-06 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/06jul25
2025-07-06 10:13:25,046 INFO [make_plots] running in process 1374216
2025-07-06 10:13:25,048 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:13:25,049 INFO [make_plots] publishing log messages to tcp://*:5565
2025-07-06 10:13:25,052 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:13:25,527 INFO [make_plots] preparing render list for 2025-07-06 NEMO forecast publish figures
2025-07-06 10:13:25,661 INFO [make_plots] added Website_thumbnail figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,664 INFO [make_plots] added Threshold_website figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,665 INFO [make_plots] added PA_tidal_predictions figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,666 INFO [make_plots] added Boundary Bay figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,667 INFO [make_plots] added Campbell River figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,668 INFO [make_plots] added Cherry Point figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,670 INFO [make_plots] added Friday Harbor figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,673 INFO [make_plots] added Halfmoon Bay figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,675 INFO [make_plots] added Nanaimo figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,676 INFO [make_plots] added Neah Bay figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,683 INFO [make_plots] added New Westminster figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,684 INFO [make_plots] added Patricia Bay figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,686 INFO [make_plots] added Point Atkinson figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,688 INFO [make_plots] added Port Renfrew figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,689 INFO [make_plots] added Sand Heads figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,691 INFO [make_plots] added Sandy Cove figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,692 INFO [make_plots] added Squamish figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,698 INFO [make_plots] added Victoria figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,699 INFO [make_plots] added Woodwards Landing figure to 2025-07-06 NEMO forecast publish render list
2025-07-06 10:13:25,700 INFO [make_plots] starting to render nemo forecast publish 06jul25 figures
2025-07-06 10:13:32,790 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-07-06 forecast run
2025-07-06 10:13:32,791 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-07-06 10:13:33,102 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-06'], host='arbutus.cloud-nowcast')
2025-07-06 10:13:35,253 INFO [run_ww3] running in process 15112
2025-07-06 10:13:35,255 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:13:35,256 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-07-06 10:13:35,257 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:13:35,260 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-07-06T101335.263499-0700
2025-07-06 10:13:35,261 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-07-06T101335.263499-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-07-06 10:13:35,271 INFO [run_ww3] nowcast WaveWatch3 run for 2025-07-06 on arbutus.cloud-nowcast started
2025-07-06 10:13:35,274 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-06 10:13:35,437 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-07-06 10:13:37,649 INFO [watch_ww3] running in process 15199
2025-07-06 10:13:37,651 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:13:37,652 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-07-06 10:13:37,657 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:13:37,697 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-06 10:16:06,200 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18554569 = 2025-07-06 01:52:00 UTC, 7.8% complete
2025-07-06 10:17:17,992 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8575379 = 2025-07-06 01:58:40 UTC, 8.2% complete
2025-07-06 10:18:37,701 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 84, 19.4% complete
2025-07-06 10:21:06,204 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18554757 = 2025-07-06 03:57:20 UTC, 16.5% complete
2025-07-06 10:21:22,625 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-07-06 10:21:22,625 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-07-06 10:21:43,646 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:22:18,432 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8575570 = 2025-07-06 04:06:00 UTC, 17.1% complete
2025-07-06 10:22:48,962 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-07-06 10:22:48,963 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-07-06 10:23:10,809 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:23:37,705 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 174, 40.3% complete
2025-07-06 10:23:51,994 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:24:09,104 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:24:11,102 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/06jul25
2025-07-06 10:24:25,474 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:24:42,238 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:24:58,469 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:25:14,890 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:25:31,257 WARNING [make_plots] invalid station number for Squamish station: None
2025-07-06 10:25:31,258 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-07-06 10:25:47,027 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:26:03,143 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-07-01 00:00:00Z to 2025-07-08 11:55:00Z
2025-07-06 10:26:06,209 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18554944 = 2025-07-06 06:02:00 UTC, 25.2% complete
2025-07-06 10:26:19,592 INFO [make_plots] finished rendering nemo forecast publish 06jul25 figures
2025-07-06 10:26:19,595 INFO [make_plots] nemo publish plots for 2025-07-06 forecast completed
2025-07-06 10:26:19,603 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-06 10:26:19,766 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:26:23,119 INFO [make_feeds] running in process 1376105
2025-07-06 10:26:23,120 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:26:23,121 INFO [make_feeds] publishing log messages to tcp://*:5565
2025-07-06 10:26:23,122 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:26:38,558 INFO [make_feeds] ATOM feeds for 2025-07-06 forecast run completed
2025-07-06 10:26:38,561 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-07-06 10:27:18,909 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8575764 = 2025-07-06 06:15:20 UTC, 26.1% complete
2025-07-06 10:28:37,709 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 264, 61.1% complete
2025-07-06 10:28:48,532 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/06jul25
2025-07-06 10:28:48,557 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-07-06 forecast into /results/nowcast-sys/figures/surface_currents/forecast/06jul25
2025-07-06 10:28:48,558 INFO [make_surface_current_tiles] surface current tile figures for 2025-07-06 forecast completed
2025-07-06 10:28:48,884 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-07-06 10:31:06,212 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18555131 = 2025-07-06 08:06:40 UTC, 33.8% complete
2025-07-06 10:32:19,385 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8575959 = 2025-07-06 08:25:20 UTC, 35.1% complete
2025-07-06 10:33:37,711 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 354, 81.9% complete
2025-07-06 10:36:06,215 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18555316 = 2025-07-06 10:10:00 UTC, 42.4% complete
2025-07-06 10:37:19,830 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8576155 = 2025-07-06 10:36:00 UTC, 44.2% complete
2025-07-06 10:38:37,718 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-06 10:41:06,306 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18555505 = 2025-07-06 12:16:00 UTC, 51.1% complete
2025-07-06 10:42:20,309 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8576350 = 2025-07-06 12:46:00 UTC, 53.2% complete
2025-07-06 10:43:37,715 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-06 10:43:37,718 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-06 10:43:38,234 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 10:43:38,241 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-06'], host='arbutus.cloud-nowcast')
2025-07-06 10:43:39,494 INFO [download_wwatch3_results] running in process 1376284
2025-07-06 10:43:39,495 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:43:39,496 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2025-07-06 10:43:39,497 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:43:40,268 INFO [run_ww3] running in process 15592
2025-07-06 10:43:40,269 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:43:40,271 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-07-06 10:43:40,272 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:43:40,276 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-07-06T104340.264861-0700
2025-07-06 10:43:40,283 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-07-06T104340.264861-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-07-06 10:43:40,288 INFO [run_ww3] forecast WaveWatch3 run for 2025-07-06 on arbutus.cloud-nowcast started
2025-07-06 10:43:40,319 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-06 10:43:40,651 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-07-06 10:43:42,828 INFO [watch_ww3] running in process 15748
2025-07-06 10:43:42,829 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 10:43:42,830 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-07-06 10:43:42,831 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-06 10:43:42,901 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-06 10:44:13,870 INFO [download_wwatch3_results] nowcast 2025-07-06 results files from arbutus.cloud-nowcast downloaded
2025-07-06 10:44:13,874 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-06 10:46:06,313 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18555693 = 2025-07-06 14:21:20 UTC, 59.8% complete
2025-07-06 10:47:20,752 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8576545 = 2025-07-06 14:56:00 UTC, 62.3% complete
2025-07-06 10:48:42,901 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 81, 12.5% complete
2025-07-06 10:51:06,220 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18555881 = 2025-07-06 16:26:40 UTC, 68.6% complete
2025-07-06 10:52:21,242 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8576738 = 2025-07-06 17:04:40 UTC, 71.2% complete
2025-07-06 10:53:42,906 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 171, 26.4% complete
2025-07-06 10:56:06,362 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18556068 = 2025-07-06 18:31:20 UTC, 77.2% complete
2025-07-06 10:57:21,683 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8576934 = 2025-07-06 19:15:20 UTC, 80.3% complete
2025-07-06 10:58:42,910 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 261, 40.3% complete
2025-07-06 11:01:06,224 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18556256 = 2025-07-06 20:36:40 UTC, 85.9% complete
2025-07-06 11:02:22,126 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8577129 = 2025-07-06 21:25:20 UTC, 89.3% complete
2025-07-06 11:03:42,899 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 345, 53.2% complete
2025-07-06 11:06:06,321 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18556445 = 2025-07-06 22:42:40 UTC, 94.7% complete
2025-07-06 11:07:22,567 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8577323 = 2025-07-06 23:34:40 UTC, 98.3% complete
2025-07-06 11:08:42,924 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 423, 65.3% complete
2025-07-06 11:11:06,226 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18556560 = 2025-07-06 23:59:20 UTC, 100.0% complete
2025-07-06 11:12:22,997 INFO [watch_NEMO_agrif] 06jul25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8577360 = 2025-07-06 23:59:20 UTC, 100.0% complete
2025-07-06 11:13:42,918 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 513, 79.2% complete
2025-07-06 11:16:06,233 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-07-06 11:16:06,238 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-06 11:16:06,821 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:16:08,207 INFO [download_results] running in process 1376644
2025-07-06 11:16:08,208 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:16:08,209 INFO [download_results] publishing log messages to tcp://*:5564
2025-07-06 11:16:08,210 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:16:08,211 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/06jul25 to /results2/SalishSea/nowcast-green.202111
2025-07-06 11:17:23,221 INFO [watch_NEMO_agrif] NEMO AGRIF run 11435164.orca2.ibb on orcinus-nowcast-agrif completed
2025-07-06 11:17:23,226 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2025-07-06 11:17:23,863 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:17:25,244 INFO [download_results] running in process 1376666
2025-07-06 11:17:25,246 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:17:25,247 INFO [download_results] publishing log messages to tcp://*:5563
2025-07-06 11:17:25,248 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:17:25,250 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/06jul25 to /results/SalishSea/nowcast-agrif.201702
2025-07-06 11:17:48,156 INFO [download_results] nowcast-agrif 2025-07-06 results files from orcinus-nowcast-agrif downloaded
2025-07-06 11:17:48,162 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-06 11:17:48,497 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:17:51,725 INFO [make_plots] running in process 1376678
2025-07-06 11:17:51,727 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:17:51,728 INFO [make_plots] publishing log messages to tcp://*:5563
2025-07-06 11:17:51,729 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:17:52,110 INFO [make_plots] preparing render list for 2025-07-06 NEMO nowcast-agrif research figures
2025-07-06 11:17:52,111 INFO [make_plots] added baynes_sound_surface figure to 2025-07-06 NEMO nowcast-agrif research render list
2025-07-06 11:17:52,112 INFO [make_plots] starting to render nemo nowcast-agrif research 06jul25 figures
2025-07-06 11:18:02,166 INFO [make_plots] finished rendering nemo nowcast-agrif research 06jul25 figures
2025-07-06 11:18:02,170 INFO [make_plots] nemo research plots for 2025-07-06 nowcast-agrif completed
2025-07-06 11:18:02,173 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-06 11:18:42,917 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 606, 93.5% complete
2025-07-06 11:23:42,924 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-07-06 11:23:42,927 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-07-06 11:23:43,454 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:23:44,751 INFO [download_wwatch3_results] running in process 1376735
2025-07-06 11:23:44,753 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:23:44,754 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-07-06 11:23:44,755 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:24:41,494 INFO [download_wwatch3_results] forecast 2025-07-06 results files from arbutus.cloud-nowcast downloaded
2025-07-06 11:24:41,498 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-07-06 11:24:41,827 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:24:43,028 INFO [update_forecast_datasets] running in process 1376746
2025-07-06 11:24:43,031 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:24:43,032 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-07-06 11:24:43,033 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:24:43,034 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/06jul25 run
2025-07-06 11:24:43,037 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/06jul25 run
2025-07-06 11:24:43,038 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/06jul25 run
2025-07-06 11:24:43,091 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/06jul25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-07-06 11:24:43,092 INFO [update_forecast_datasets] wwatch3 2025-07-06 forecast rolling forecast datasets updated
2025-07-06 11:24:43,095 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-06 11:24:43,433 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-07-06 11:24:44,623 INFO [ping_erddap] running in process 1376751
2025-07-06 11:24:44,625 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:24:44,626 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-06 11:24:44,627 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:24:44,629 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-07-06 11:24:44,631 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-06 11:24:44,954 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:24:48,598 INFO [make_plots] running in process 1376757
2025-07-06 11:24:48,599 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:24:48,601 INFO [make_plots] publishing log messages to tcp://*:5563
2025-07-06 11:24:48,602 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:24:48,603 INFO [make_plots] preparing render list for 2025-07-06 WaveWatch3 forecast2 publish figures
2025-07-06 11:24:48,604 INFO [make_plots] added Halibut Bank figure to 2025-07-06 WaveWatch3 forecast2 publish render list
2025-07-06 11:24:48,605 INFO [make_plots] added Sentry Shoal figure to 2025-07-06 WaveWatch3 forecast2 publish render list
2025-07-06 11:24:48,606 INFO [make_plots] starting to render wwatch3 forecast2 publish 06jul25 figures
2025-07-06 11:24:49,057 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-07-06 11:24:54,624 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-07-06 11:25:00,048 INFO [make_plots] finished rendering wwatch3 forecast2 publish 06jul25 figures
2025-07-06 11:25:00,049 INFO [make_plots] wwatch3 publish plots for 2025-07-06 forecast2 completed
2025-07-06 11:25:00,053 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-06 11:29:44,660 INFO [download_results] nowcast-green 2025-07-06 results files from arbutus.cloud-nowcast downloaded
2025-07-06 11:29:44,667 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-06 11:29:45,006 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:29:45,012 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-07-06 11:29:45,017 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:29:45,023 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:29:45,028 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-07-06'], host='localhost')
2025-07-06 11:29:46,354 INFO [ping_erddap] running in process 1376845
2025-07-06 11:29:46,356 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:29:46,356 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-06 11:29:46,357 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:29:46,361 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-07-06 11:29:46,365 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-06 11:29:47,400 INFO [make_averaged_dataset] running in process 1376846
2025-07-06 11:29:47,401 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:29:47,402 INFO [make_averaged_dataset] publishing log messages to tcp://*:5563
2025-07-06 11:29:47,403 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:29:47,404 INFO [make_averaged_dataset] creating day-averaged dataset for 06-Jul-2025 biology
2025-07-06 11:29:47,405 INFO [make_averaged_dataset] running in process 1376847
2025-07-06 11:29:47,406 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:29:47,407 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-07-06 11:29:47,408 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:29:47,409 INFO [make_averaged_dataset] creating day-averaged dataset for 06-Jul-2025 physics
2025-07-06 11:29:47,422 INFO [make_averaged_dataset] running in process 1376848
2025-07-06 11:29:47,423 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:29:47,423 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-07-06 11:29:47,424 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:29:47,425 INFO [make_averaged_dataset] creating day-averaged dataset for 06-Jul-2025 chemistry
2025-07-06 11:29:48,003 INFO [make_plots] running in process 1376843
2025-07-06 11:29:48,004 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 11:29:48,005 INFO [make_plots] publishing log messages to tcp://*:5566
2025-07-06 11:29:48,005 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-06 11:29:48,304 INFO [make_plots] preparing render list for 2025-07-06 NEMO nowcast-green research figures
2025-07-06 11:29:50,125 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:50,960 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:51,740 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:52,530 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:53,278 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:54,023 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:54,782 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:55,626 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:56,465 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:57,311 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:58,161 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:29:58,911 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:30:00,283 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:30:00,286 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:30:00,287 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:30:00,288 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-07-06 NEMO nowcast-green research render list
2025-07-06 11:30:00,288 INFO [make_plots] starting to render nemo nowcast-green research 06jul25 figures
2025-07-06 11:30:20,304 INFO [make_averaged_dataset] day-averaged dataset for 06-Jul-2025 physics created
2025-07-06 11:30:20,308 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-06 11:30:56,091 INFO [make_averaged_dataset] day-averaged dataset for 06-Jul-2025 chemistry created
2025-07-06 11:30:56,095 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-06 11:31:12,954 INFO [make_averaged_dataset] day-averaged dataset for 06-Jul-2025 biology created
2025-07-06 11:31:12,957 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-07-06 12:34:21,585 INFO [make_plots] finished rendering nemo nowcast-green research 06jul25 figures
2025-07-06 12:34:21,758 INFO [make_plots] nemo research plots for 2025-07-06 nowcast-green completed
2025-07-06 12:34:21,825 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-06 15:03:16,738 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250706/18/
2025-07-06 15:03:16,740 INFO [collect_weather] 2025-07-06 2.5km weather forecast 18 collection complete
2025-07-06 15:03:16,743 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-06 15:03:17,262 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-07-06 15:03:17,273 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-07-06 15:03:17,277 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-07-06 15:03:17,282 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-07-07'], host='localhost')
2025-07-06 15:03:18,546 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250706/18/
2025-07-06 15:03:18,547 INFO [crop_gribs] 2025-07-06 18 GRIBs cropping complete
2025-07-06 15:03:18,549 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-06 15:03:18,907 INFO [download_weather] running in process 1378886
2025-07-06 15:03:18,908 INFO [collect_weather] running in process 1378887
2025-07-06 15:03:18,909 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 15:03:18,910 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 15:03:18,911 INFO [download_weather] publishing log messages to tcp://*:5562
2025-07-06 15:03:18,912 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-07-06 15:03:18,913 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-06 15:03:18,914 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-06 15:03:18,915 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250706
2025-07-06 15:03:18,920 INFO [download_weather] running in process 1378884
2025-07-06 15:03:18,921 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 15:03:18,922 INFO [download_weather] publishing log messages to tcp://*:5564
2025-07-06 15:03:18,923 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-06 15:03:18,923 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250706
2025-07-06 15:03:18,929 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-07-06 15:03:19,844 INFO [crop_gribs] running in process 1378888
2025-07-06 15:03:19,845 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 15:03:19,846 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-07-06 15:03:19,848 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-06 15:03:19,849 INFO [crop_gribs] cropping 2025-07-07 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-07-06 15:03:19,860 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250707/00/
2025-07-06 15:07:05,330 INFO [download_weather] 2025-07-06 1km weather forecast 12 downloads complete
2025-07-06 15:07:05,334 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-07-06 15:07:05,626 INFO [download_weather] 2025-07-06 1km weather forecast 00 downloads complete
2025-07-06 15:07:05,893 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-07-06 21:03:11,698 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/00/ to /results/forcing/atmospheric/continental2.5/GRIB/20250707/00/
2025-07-06 21:03:11,699 INFO [collect_weather] 2025-07-07 2.5km weather forecast 00 collection complete
2025-07-06 21:03:11,702 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-06 21:03:12,244 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['06', '2.5km'], host='localhost')
2025-07-06 21:03:12,249 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['06', '--fcst-date', '2025-07-07'], host='localhost')
2025-07-06 21:03:12,571 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250707/00/
2025-07-06 21:03:12,572 INFO [crop_gribs] 2025-07-07 00 GRIBs cropping complete
2025-07-06 21:03:12,574 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-06 21:03:13,530 INFO [collect_weather] running in process 1382863
2025-07-06 21:03:13,532 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 21:03:13,533 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-07-06 21:03:13,535 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-06 21:03:14,291 INFO [crop_gribs] running in process 1382865
2025-07-06 21:03:14,293 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-06 21:03:14,293 INFO [crop_gribs] publishing log messages to tcp://*:5563
2025-07-06 21:03:14,295 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-06 21:03:14,296 INFO [crop_gribs] cropping 2025-07-07 ECCC HRDPS 2.5km continental 06Z GRIB files to SalishSeaCast subdomain
2025-07-06 21:03:14,311 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250707/06/
2025-07-06 21:03:16,627 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/06/
2025-07-07 03:08:29,486 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/06/ to /results/forcing/atmospheric/continental2.5/GRIB/20250707/06/
2025-07-07 03:08:29,487 INFO [collect_weather] 2025-07-07 2.5km weather forecast 06 collection complete
2025-07-07 03:08:29,490 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-07-07 03:08:29,630 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250707/06/
2025-07-07 03:08:29,631 INFO [crop_gribs] 2025-07-07 06 GRIBs cropping complete
2025-07-07 03:08:30,094 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Capilano', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,102 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ChilliwackVedder', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,107 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'ClowhomClowhomLake', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,113 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Englishman', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,118 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'Fraser', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,123 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'HomathkoMouth', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,127 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'NicomeklLangley', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,132 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'RobertsCreek', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,136 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SalmonSayward', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,145 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SanJuanPortRenfrew', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,157 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'SquamishBrackendale', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,169 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaScotty', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,181 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaBypass', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,197 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['ECCC', 'TheodosiaDiversion', '--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,214 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SCVIP'], host='localhost')
2025-07-07 03:08:30,236 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ctd', args=['SEVIP'], host='localhost')
2025-07-07 03:08:30,250 INFO [manager] launching NextWorker(module='nowcast.workers.get_onc_ferry', args=['TWDP'], host='localhost')
2025-07-07 03:08:30,257 INFO [manager] launching NextWorker(module='nowcast.workers.get_vfpa_hadcp', args=['--data-date', '2025-07-06'], host='localhost')
2025-07-07 03:08:30,273 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['00'], host='localhost')
2025-07-07 03:08:30,284 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['12', '2.5km'], host='localhost')
2025-07-07 03:08:30,295 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['12'], host='localhost')
2025-07-07 03:08:30,325 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-07-07 03:08:31,623 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['forecast2'], host='localhost')
2025-07-07 03:08:33,464 INFO [collect_weather] running in process 1387202
2025-07-07 03:08:33,465 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:33,466 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-07-07 03:08:33,466 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:33,480 INFO [collect_NeahBay_ssh] running in process 1387201
2025-07-07 03:08:33,481 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:33,482 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-07-07 03:08:33,487 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:33,490 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 00Z obs/forecast for 2025-07-07
2025-07-07 03:08:33,493 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/12/
2025-07-07 03:08:34,340 INFO [collect_river_data] running in process 1387190
2025-07-07 03:08:34,341 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,341 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-07-07 03:08:34,342 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,343 INFO [collect_river_data] Collecting ECCC RobertsCreek river data for 2025-07-06
2025-07-07 03:08:34,343 INFO [collect_river_data] running in process 1387191
2025-07-07 03:08:34,344 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,346 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-07-07 03:08:34,346 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,347 INFO [collect_river_data] Collecting ECCC SalmonSayward river data for 2025-07-06
2025-07-07 03:08:34,352 INFO [collect_river_data] running in process 1387181
2025-07-07 03:08:34,353 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,354 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-07-07 03:08:34,355 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,355 INFO [collect_river_data] Collecting ECCC Capilano river data for 2025-07-06
2025-07-07 03:08:34,374 INFO [collect_river_data] Appended ECCC SalmonSayward river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Salmon_Sayward_flow
2025-07-07 03:08:34,374 INFO [collect_river_data] ECCC SalmonSayward river data collection for 2025-07-06 completed
2025-07-07 03:08:34,376 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:34,378 INFO [collect_river_data] Appended ECCC RobertsCreek river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/RobertsCreek_flow
2025-07-07 03:08:34,379 INFO [collect_river_data] ECCC RobertsCreek river data collection for 2025-07-06 completed
2025-07-07 03:08:34,389 INFO [collect_river_data] running in process 1387187
2025-07-07 03:08:34,390 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,391 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-07-07 03:08:34,391 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,392 INFO [collect_river_data] Collecting ECCC HomathkoMouth river data for 2025-07-06
2025-07-07 03:08:34,412 INFO [collect_river_data] Appended ECCC Capilano river average discharge for 2025-07-06 to: /opp/observations/rivers/Capilano/Caplilano_08GA010_day_avg_flow
2025-07-07 03:08:34,414 INFO [collect_river_data] ECCC Capilano river data collection for 2025-07-06 completed
2025-07-07 03:08:34,421 INFO [collect_river_data] Appended ECCC HomathkoMouth river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Homathko_Mouth_flow
2025-07-07 03:08:34,422 INFO [collect_river_data] ECCC HomathkoMouth river data collection for 2025-07-06 completed
2025-07-07 03:08:34,445 INFO [collect_river_data] running in process 1387186
2025-07-07 03:08:34,446 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,446 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-07-07 03:08:34,447 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,448 INFO [collect_river_data] Collecting ECCC Fraser river data for 2025-07-06
2025-07-07 03:08:34,451 INFO [collect_river_data] running in process 1387188
2025-07-07 03:08:34,452 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,454 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-07-07 03:08:34,455 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,456 INFO [collect_river_data] Collecting ECCC NicomeklLangley river data for 2025-07-06
2025-07-07 03:08:34,467 INFO [collect_river_data] running in process 1387185
2025-07-07 03:08:34,468 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,468 INFO [collect_river_data] publishing log messages to tcp://*:5570
2025-07-07 03:08:34,469 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,470 INFO [collect_river_data] Collecting ECCC Englishman river data for 2025-07-06
2025-07-07 03:08:34,476 INFO [collect_river_data] running in process 1387195
2025-07-07 03:08:34,476 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,477 INFO [collect_river_data] publishing log messages to tcp://*:5571
2025-07-07 03:08:34,478 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,479 INFO [collect_river_data] Collecting ECCC TheodosiaBypass river data for 2025-07-06
2025-07-07 03:08:34,480 INFO [collect_river_data] Appended ECCC Fraser river average discharge for 2025-07-06 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Fraser_flow
2025-07-07 03:08:34,480 INFO [collect_river_data] ECCC Fraser river data collection for 2025-07-06 completed
2025-07-07 03:08:34,491 INFO [collect_river_data] Appended ECCC NicomeklLangley river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Nicomekl_Langley_flow
2025-07-07 03:08:34,492 INFO [collect_river_data] ECCC NicomeklLangley river data collection for 2025-07-06 completed
2025-07-07 03:08:34,495 INFO [collect_river_data] Appended ECCC Englishman river average discharge for 2025-07-06 to: /data/dlatorne/SOG-projects/SOG-forcing/ECget/Englishman_flow
2025-07-07 03:08:34,495 INFO [collect_river_data] ECCC Englishman river data collection for 2025-07-06 completed
2025-07-07 03:08:34,509 INFO [collect_river_data] running in process 1387183
2025-07-07 03:08:34,509 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,510 INFO [collect_river_data] publishing log messages to tcp://*:5572
2025-07-07 03:08:34,511 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,512 INFO [collect_river_data] Collecting ECCC ChilliwackVedder river data for 2025-07-06
2025-07-07 03:08:34,513 INFO [collect_river_data] running in process 1387193
2025-07-07 03:08:34,514 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,515 INFO [collect_river_data] publishing log messages to tcp://*:5573
2025-07-07 03:08:34,516 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,517 INFO [collect_river_data] Collecting ECCC SquamishBrackendale river data for 2025-07-06
2025-07-07 03:08:34,518 INFO [collect_river_data] Appended ECCC TheodosiaBypass river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Theodosia_Bypass_flow
2025-07-07 03:08:34,518 INFO [collect_river_data] ECCC TheodosiaBypass river data collection for 2025-07-06 completed
2025-07-07 03:08:34,519 INFO [collect_river_data] running in process 1387194
2025-07-07 03:08:34,520 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,521 INFO [collect_river_data] publishing log messages to tcp://*:5574
2025-07-07 03:08:34,522 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,522 INFO [collect_river_data] Collecting ECCC TheodosiaScotty river data for 2025-07-06
2025-07-07 03:08:34,523 INFO [collect_river_data] running in process 1387192
2025-07-07 03:08:34,523 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,524 INFO [collect_river_data] publishing log messages to tcp://*:5575
2025-07-07 03:08:34,525 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,525 INFO [collect_river_data] Collecting ECCC SanJuanPortRenfrew river data for 2025-07-06
2025-07-07 03:08:34,541 INFO [collect_river_data] Appended ECCC SquamishBrackendale river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Squamish_Brackendale_flow
2025-07-07 03:08:34,542 INFO [collect_river_data] ECCC SquamishBrackendale river data collection for 2025-07-06 completed
2025-07-07 03:08:34,546 INFO [collect_river_data] Appended ECCC TheodosiaScotty river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Theodosia_Scotty_flow
2025-07-07 03:08:34,547 INFO [collect_river_data] ECCC TheodosiaScotty river data collection for 2025-07-06 completed
2025-07-07 03:08:34,562 INFO [collect_river_data] Appended ECCC ChilliwackVedder river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Chilliwack_Vedder_flow
2025-07-07 03:08:34,563 INFO [collect_river_data] ECCC ChilliwackVedder river data collection for 2025-07-06 completed
2025-07-07 03:08:34,564 INFO [collect_river_data] Appended ECCC SanJuanPortRenfrew river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/SanJuan_PortRenfrew_flow
2025-07-07 03:08:34,565 INFO [collect_river_data] ECCC SanJuanPortRenfrew river data collection for 2025-07-06 completed
2025-07-07 03:08:34,585 INFO [collect_river_data] running in process 1387184
2025-07-07 03:08:34,585 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,586 INFO [collect_river_data] publishing log messages to tcp://*:5576
2025-07-07 03:08:34,587 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,588 INFO [collect_river_data] Collecting ECCC ClowhomClowhomLake river data for 2025-07-06
2025-07-07 03:08:34,600 INFO [collect_river_data] running in process 1387196
2025-07-07 03:08:34,603 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,604 INFO [collect_river_data] publishing log messages to tcp://*:5577
2025-07-07 03:08:34,604 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,605 INFO [collect_river_data] Collecting ECCC TheodosiaDiversion river data for 2025-07-06
2025-07-07 03:08:34,621 INFO [collect_river_data] Appended ECCC ClowhomClowhomLake river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Clowhom_ClowhomLake_flow
2025-07-07 03:08:34,622 INFO [collect_river_data] ECCC ClowhomClowhomLake river data collection for 2025-07-06 completed
2025-07-07 03:08:34,635 INFO [collect_river_data] Appended ECCC TheodosiaDiversion river average discharge for 2025-07-06 to: /results/forcing/rivers/observations/Theodosia_Diversion_flow
2025-07-07 03:08:34,636 INFO [collect_river_data] ECCC TheodosiaDiversion river data collection for 2025-07-06 completed
2025-07-07 03:08:34,703 INFO [get_vfpa_hadcp] running in process 1387200
2025-07-07 03:08:34,704 INFO [get_vfpa_hadcp] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,705 INFO [get_vfpa_hadcp] publishing log messages to tcp://*:5578
2025-07-07 03:08:34,706 INFO [get_vfpa_hadcp] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,707 INFO [get_vfpa_hadcp] processing VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-07-06
2025-07-07 03:08:34,792 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:34,944 INFO [crop_gribs] running in process 1387203
2025-07-07 03:08:34,945 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:34,946 INFO [crop_gribs] publishing log messages to tcp://*:5579
2025-07-07 03:08:34,946 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:34,947 INFO [crop_gribs] cropping 2025-07-07 ECCC HRDPS 2.5km continental 12Z GRIB files to SalishSeaCast subdomain
2025-07-07 03:08:34,961 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250707/12/
2025-07-07 03:08:35,123 INFO [grib_to_netcdf] running in process 1387407
2025-07-07 03:08:35,128 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:35,128 INFO [grib_to_netcdf] publishing log messages to tcp://*:5565
2025-07-07 03:08:35,129 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:35,130 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-07-07 forecast2 run
2025-07-07 03:08:35,134 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:35,256 INFO [get_onc_ctd] running in process 1387198
2025-07-07 03:08:35,257 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:35,258 INFO [get_onc_ctd] publishing log messages to tcp://*:5590
2025-07-07 03:08:35,259 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:35,260 INFO [get_onc_ctd] requesting ONC SEVIP CTD T&S data for 2025-07-06
2025-07-07 03:08:35,261 INFO [get_onc_ferry] running in process 1387199
2025-07-07 03:08:35,264 INFO [get_onc_ferry] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:35,267 INFO [get_onc_ferry] publishing log messages to tcp://*:5591
2025-07-07 03:08:35,267 INFO [get_onc_ferry] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:35,268 INFO [get_onc_ferry] requesting ONC TWDP.N1 NAV data for 2025-07-06
2025-07-07 03:08:35,272 INFO [get_onc_ctd] running in process 1387197
2025-07-07 03:08:35,273 INFO [get_onc_ctd] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:35,273 INFO [get_onc_ctd] publishing log messages to tcp://*:5592
2025-07-07 03:08:35,274 INFO [get_onc_ctd] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:35,275 INFO [get_onc_ctd] requesting ONC SCVIP CTD T&S data for 2025-07-06
2025-07-07 03:08:35,468 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:35,822 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:35,869 INFO [collect_NeahBay_ssh] 2025-07-07T00:00:00-07:00 Neah Bay ssh 00Z obs/forecast data collection complete
2025-07-07 03:08:36,150 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:36,476 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:36,718 INFO [get_vfpa_hadcp] added VFPA HADCP data from 2nd Narrows Rail Bridge for 2025-07-06 to /opp/observations/AISDATA/netcdf/VFPA_2ND_NARROWS_HADCP_2s_202507.nc
2025-07-07 03:08:36,719 INFO [get_vfpa_hadcp] VFPA HADCP observations added to 2025-07 netcdf file
2025-07-07 03:08:36,800 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:37,102 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:37,408 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:37,726 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:38,050 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:38,384 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:38,735 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-07-07 03:08:39,076 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-07-07 03:08:39,436 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['forecast2', '--run-date', '2025-07-07'], host='localhost')
2025-07-07 03:08:39,442 INFO [manager] checklist updated with [VFPA HADCP data] items from get_vfpa_hadcp worker
2025-07-07 03:08:39,965 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['VFPA-HADCP'], host='localhost')
2025-07-07 03:08:41,101 INFO [ping_erddap] running in process 1387547
2025-07-07 03:08:41,103 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:41,104 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-07 03:08:41,105 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:41,107 INFO [ping_erddap] VFPA-HADCP ERDDAP dataset flag file(s) created
2025-07-07 03:08:41,108 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-07 03:08:42,689 INFO [make_ssh_files] running in process 1387534
2025-07-07 03:08:42,690 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:42,691 INFO [make_ssh_files] publishing log messages to tcp://*:5563
2025-07-07 03:08:42,692 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:42,693 INFO [make_ssh_files] building forecast2 sea surface height boundary conditions file(s) from 2025-07-07 NOAA Neah Bay 00Z observation and forecast values
2025-07-07 03:08:44,741 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d08.nc
2025-07-07 03:08:45,458 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d02.nc
2025-07-07 03:08:45,736 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d03.nc
2025-07-07 03:08:46,014 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d04.nc
2025-07-07 03:08:46,293 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d05.nc
2025-07-07 03:08:46,556 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m07d06.nc
2025-07-07 03:08:46,850 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d07.nc
2025-07-07 03:08:47,041 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d08.nc
2025-07-07 03:08:47,282 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d09.nc
2025-07-07 03:08:47,520 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d10.nc
2025-07-07 03:08:47,636 INFO [make_ssh_files] sea surface height boundary file for 2025-07-07 forecast2 run created
2025-07-07 03:08:47,639 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-07-07 03:08:47,832 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d09.nc
2025-07-07 03:08:47,839 INFO [grib_to_netcdf] 2025-07-07 NEMO-atmos forcing file for forecast2 created
2025-07-07 03:08:48,014 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-07-07 03:08:48,020 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-07-07 03:08:48,026 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-07-07 03:08:49,425 INFO [make_201702_runoff_file] running in process 1387582
2025-07-07 03:08:49,426 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:49,427 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5563
2025-07-07 03:08:49,428 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:49,462 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-07-07 03:08:49,578 INFO [make_201702_runoff_file] 2025-07-07 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-07-07 03:08:49,581 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-07-07 03:08:49,974 INFO [make_runoff_file] running in process 1387580
2025-07-07 03:08:49,976 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:49,977 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-07-07 03:08:49,978 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:49,980 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-07-06
2025-07-07 03:08:51,090 ERROR [make_runoff_file] no 2025-07-06 discharge obs for SkagitMountVernon: patching
2025-07-07 03:08:51,742 ERROR [make_runoff_file] no 2025-07-06 discharge obs for SnohomishMonroe: patching
2025-07-07 03:08:51,984 ERROR [make_runoff_file] no 2025-07-06 discharge obs for NisquallyMcKenna: patching
2025-07-07 03:08:52,087 ERROR [make_runoff_file] no 2025-07-06 discharge obs for GreenwaterGreenwater: patching
2025-07-07 03:08:53,388 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-07-06: /results/forcing/rivers/R202108Dailies_y2025m07d06.nc
2025-07-07 03:08:53,389 INFO [make_runoff_file] 2025-07-06 runoff file creation completed
2025-07-07 03:08:53,392 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-07-07 03:08:53,713 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'forecast2'], host='localhost')
2025-07-07 03:08:53,716 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'forecast2'], host='localhost')
2025-07-07 03:08:53,723 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'forecast2'], host='localhost')
2025-07-07 03:08:53,728 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'forecast2'], host='localhost')
2025-07-07 03:08:54,554 INFO [get_onc_ctd] 2025-07-06 ONC SCVIP CTD T&S file created
2025-07-07 03:08:54,556 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-07-07 03:08:54,928 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SCVIP-CTD'], host='localhost')
2025-07-07 03:08:55,067 INFO [upload_forcing] running in process 1387615
2025-07-07 03:08:55,068 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:55,069 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-07-07 03:08:55,070 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:55,078 INFO [upload_forcing] running in process 1387618
2025-07-07 03:08:55,080 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:55,081 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-07-07 03:08:55,081 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:55,109 INFO [upload_forcing] running in process 1387617
2025-07-07 03:08:55,110 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:55,111 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-07-07 03:08:55,112 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:55,145 INFO [upload_forcing] running in process 1387616
2025-07-07 03:08:55,146 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:55,147 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-07-07 03:08:55,148 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:55,306 INFO [get_onc_ctd] 2025-07-06 ONC SEVIP CTD T&S file created
2025-07-07 03:08:55,308 INFO [manager] checklist updated with [ONC CTD data] items from get_onc_ctd worker
2025-07-07 03:08:55,710 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['SEVIP-CTD'], host='localhost')
2025-07-07 03:08:56,221 INFO [ping_erddap] running in process 1387632
2025-07-07 03:08:56,229 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:56,236 INFO [ping_erddap] publishing log messages to tcp://*:5567
2025-07-07 03:08:56,246 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:56,251 INFO [ping_erddap] SCVIP-CTD ERDDAP dataset flag file(s) created
2025-07-07 03:08:56,254 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-07 03:08:56,592 INFO [upload_forcing] LiveOcean boundary conditions file not found; created symlink to /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m07d06.nc
2025-07-07 03:08:56,641 INFO [upload_forcing] forecast2 2025-07-07 forcing files upload to optimum-hindcast completed
2025-07-07 03:08:56,647 INFO [upload_forcing] forecast2 2025-07-07 forcing files upload to orcinus-nowcast-agrif completed
2025-07-07 03:08:56,650 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-07 03:08:56,885 INFO [get_onc_ferry] requesting ONC TWDP TSG data for 2025-07-06
2025-07-07 03:08:56,974 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-07 03:08:57,293 INFO [ping_erddap] running in process 1387647
2025-07-07 03:08:57,294 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:08:57,295 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-07-07 03:08:57,296 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:08:57,297 INFO [ping_erddap] SEVIP-CTD ERDDAP dataset flag file(s) created
2025-07-07 03:08:57,326 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-07 03:09:04,152 INFO [upload_forcing] forecast2 2025-07-07 forcing files upload to robot.graham completed
2025-07-07 03:09:04,156 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-07 03:09:05,361 INFO [upload_forcing] forecast2 2025-07-07 forcing files upload to arbutus.cloud-nowcast completed
2025-07-07 03:09:05,366 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-07-07 03:09:05,732 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-07'], host='localhost')
2025-07-07 03:09:07,034 INFO [make_forcing_links] running in process 1387663
2025-07-07 03:09:07,035 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:09:07,036 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-07-07 03:09:07,037 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:09:09,364 INFO [make_forcing_links] forecast2 2025-07-07 forcing file links on arbutus.cloud-nowcast created
2025-07-07 03:09:09,368 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-07-07 03:09:09,728 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-07'], host='arbutus.cloud-nowcast')
2025-07-07 03:09:12,670 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-07-07 03:09:12,977 INFO [run_NEMO] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/runs/06jul25forecast2_2025-07-07T030912.035443-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-07-07 03:09:12,988 INFO [run_NEMO] forecast2 NEMO run for 2025-07-07 on arbutus.cloud-nowcast started
2025-07-07 03:09:12,991 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-07-07 03:09:13,356 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-07-07 03:09:15,397 INFO [watch_NEMO] running in process 26152
2025-07-07 03:09:15,399 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:09:15,400 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-07-07 03:09:15,401 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:09:15,532 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-07-07 03:09:19,992 INFO [get_onc_ferry] requesting ONC TWDP OXYSENSOR data for 2025-07-06
2025-07-07 03:09:42,679 INFO [get_onc_ferry] requesting ONC TWDP TURBCHLFL data for 2025-07-06
2025-07-07 03:10:09,008 INFO [get_onc_ferry] requesting ONC TWDP CO2SENSOR data for 2025-07-06
2025-07-07 03:10:09,045 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure,co2 data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,050 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR partial_pressure data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,052 WARNING [get_onc_ferry] No ONC TWDP CO2SENSOR co2 data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,053 INFO [get_onc_ferry] requesting ONC TWDP TEMPHUMID data for 2025-07-06
2025-07-07 03:10:09,100 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature,rel_humidity data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,104 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID air_temperature data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,106 WARNING [get_onc_ferry] No ONC TWDP TEMPHUMID rel_humidity data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,107 INFO [get_onc_ferry] requesting ONC TWDP BARPRESS data for 2025-07-06
2025-07-07 03:10:09,188 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,190 WARNING [get_onc_ferry] No ONC TWDP BARPRESS barometric_pressure data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,191 INFO [get_onc_ferry] requesting ONC TWDP PYRANOMETER data for 2025-07-06
2025-07-07 03:10:09,232 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,235 WARNING [get_onc_ferry] No ONC TWDP PYRANOMETER solar_radiation data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,236 INFO [get_onc_ferry] requesting ONC TWDP PYRGEOMETER data for 2025-07-06
2025-07-07 03:10:09,285 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,287 WARNING [get_onc_ferry] No ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-06; substituting empty dataset
2025-07-07 03:10:09,479 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR partial_pressure data for 2025-07-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-07 03:10:09,494 WARNING [get_onc_ferry] ONC TWDP CO2SENSOR co2 data for 2025-07-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-07 03:10:09,510 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID air_temperature data for 2025-07-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-07 03:10:09,526 WARNING [get_onc_ferry] ONC TWDP TEMPHUMID rel_humidity data for 2025-07-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-07 03:10:09,541 WARNING [get_onc_ferry] ONC TWDP BARPRESS barometric_pressure data for 2025-07-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-07 03:10:09,558 WARNING [get_onc_ferry] ONC TWDP PYRANOMETER solar_radiation data for 2025-07-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-07 03:10:09,573 WARNING [get_onc_ferry] ONC TWDP PYRGEOMETER downward_radiation data for 2025-07-06 contains no qaqcFlag<=1 or qaqcFlac>=7 values; substituting NaNs
2025-07-07 03:11:11,635 INFO [get_onc_ferry] 2025-07-06 ONC TWDP ferry data file created
2025-07-07 03:11:11,638 INFO [manager] checklist updated with [ONC ferry data] items from get_onc_ferry worker
2025-07-07 03:11:11,956 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['TWDP-ferry'], host='localhost')
2025-07-07 03:11:13,215 INFO [ping_erddap] running in process 1387690
2025-07-07 03:11:13,216 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:11:13,217 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-07 03:11:13,219 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:11:13,221 INFO [ping_erddap] TWDP-ferry ERDDAP dataset flag file(s) created
2025-07-07 03:11:13,223 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-07 03:14:15,532 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18559184 = 2025-07-08 05:08:40 UTC, 17.2% complete
2025-07-07 03:19:15,532 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18559680 = 2025-07-08 10:39:20 UTC, 35.5% complete
2025-07-07 03:24:15,603 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18560174 = 2025-07-08 16:08:40 UTC, 53.8% complete
2025-07-07 03:29:15,535 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18560660 = 2025-07-08 21:32:40 UTC, 71.8% complete
2025-07-07 03:34:15,540 INFO [watch_NEMO] forecast2 on arbutus.cloud-nowcast: timestep: 18561158 = 2025-07-09 03:04:40 UTC, 90.3% complete
2025-07-07 03:39:15,588 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-07-07 03:39:15,631 INFO [watch_NEMO] forecast2 NEMO run on arbutus.cloud-nowcast completed
2025-07-07 03:39:16,204 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-07'], host='arbutus.cloud-nowcast')
2025-07-07 03:39:16,212 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-07'], host='arbutus.cloud-nowcast')
2025-07-07 03:39:16,217 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-06'], host='localhost')
2025-07-07 03:39:17,555 INFO [download_results] running in process 1387930
2025-07-07 03:39:17,556 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:39:17,557 INFO [download_results] publishing log messages to tcp://*:5563
2025-07-07 03:39:17,558 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:39:17,560 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast2/06jul25 to /results/SalishSea/forecast2.202111
2025-07-07 03:39:18,885 INFO [make_ww3_current_file] running in process 26527
2025-07-07 03:39:18,886 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:39:18,887 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-07-07 03:39:18,888 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:39:18,889 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-07-07 forecast2 run
2025-07-07 03:39:19,098 INFO [make_ww3_wind_file] running in process 26529
2025-07-07 03:39:19,099 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:39:19,101 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-07-07 03:39:19,102 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:39:19,103 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-07-07 forecast2 run
2025-07-07 03:39:27,759 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-07-07 forecast2 run
2025-07-07 03:39:27,763 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-07-07 03:42:58,713 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-07-07 forecast2 run
2025-07-07 03:42:58,719 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-07-07 03:42:59,078 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-07-07'], host='arbutus.cloud-nowcast')
2025-07-07 03:43:01,169 INFO [run_ww3] running in process 27086
2025-07-07 03:43:01,170 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:43:01,171 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-07-07 03:43:01,173 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:43:01,177 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-07-07T034301.168432-0700
2025-07-07 03:43:01,178 INFO [run_ww3] forecast2: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast2_2025-07-07T034301.168432-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-07-07 03:43:01,194 INFO [run_ww3] forecast2 WaveWatch3 run for 2025-07-07 on arbutus.cloud-nowcast started
2025-07-07 03:43:01,214 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-07-07 03:43:01,585 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast2'], host='arbutus.cloud-nowcast')
2025-07-07 03:43:03,794 INFO [watch_ww3] running in process 27173
2025-07-07 03:43:03,796 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:43:03,797 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-07-07 03:43:03,798 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:43:03,824 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-07-07 03:43:37,618 INFO [download_results] forecast2 2025-07-06 results files from arbutus.cloud-nowcast downloaded
2025-07-07 03:43:37,623 INFO [manager] checklist updated with [results files] items from download_results worker
2025-07-07 03:43:37,979 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast2', '--run-date', '2025-07-06'], host='localhost')
2025-07-07 03:43:40,219 INFO [make_CHS_currents_file] running in process 1387979
2025-07-07 03:43:40,220 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:43:40,222 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-07-07 03:43:40,223 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:44:19,295 INFO [make_CHS_currents_file] Made CHS currents file for 2025-07-06 for forecast2
2025-07-07 03:44:19,298 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-07-07 03:44:19,666 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast2', '--run-date', '2025-07-06'], host='localhost')
2025-07-07 03:44:20,840 INFO [update_forecast_datasets] running in process 1387999
2025-07-07 03:44:20,841 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:44:20,842 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-07-07 03:44:20,843 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:44:20,845 INFO [update_forecast_datasets] updating nemo forecast directory for forecast2/06jul25 run
2025-07-07 03:48:03,776 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 99, 18.3% complete
2025-07-07 03:48:34,565 INFO [update_forecast_datasets] updated nemo forecast directory for forecast2/06jul25 run: /results/SalishSea/rolling-forecasts/nemo
2025-07-07 03:48:34,566 INFO [update_forecast_datasets] nemo 2025-07-06 forecast2 rolling forecast datasets updated
2025-07-07 03:48:34,570 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-07-07 03:48:34,936 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-07-07 03:48:34,942 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast2', 'publish', '--run-date', '2025-07-06'], host='localhost')
2025-07-07 03:48:34,947 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast2', '--run-date', '2025-07-06'], host='localhost')
2025-07-07 03:48:36,163 INFO [ping_erddap] running in process 1388100
2025-07-07 03:48:36,164 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:48:36,165 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-07-07 03:48:36,166 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:48:36,175 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-07-07 03:48:36,177 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-07-07 03:48:37,395 INFO [make_surface_current_tiles] running in process 1388103
2025-07-07 03:48:37,396 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:48:37,397 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5563
2025-07-07 03:48:37,399 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:48:37,400 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-07-06 forecast2 in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast2/06jul25
2025-07-07 03:48:38,019 INFO [make_plots] running in process 1388102
2025-07-07 03:48:38,020 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 03:48:38,022 INFO [make_plots] publishing log messages to tcp://*:5564
2025-07-07 03:48:38,023 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-07-07 03:48:38,448 INFO [make_plots] preparing render list for 2025-07-06 NEMO forecast2 publish figures
2025-07-07 03:48:38,595 INFO [make_plots] added Website_thumbnail figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,596 INFO [make_plots] added Threshold_website figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,597 INFO [make_plots] added PA_tidal_predictions figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,598 INFO [make_plots] added Boundary Bay figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,599 INFO [make_plots] added Campbell River figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,599 INFO [make_plots] added Cherry Point figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,600 INFO [make_plots] added Friday Harbor figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,601 INFO [make_plots] added Halfmoon Bay figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,601 INFO [make_plots] added Nanaimo figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,602 INFO [make_plots] added Neah Bay figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,602 INFO [make_plots] added New Westminster figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,603 INFO [make_plots] added Patricia Bay figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,603 INFO [make_plots] added Point Atkinson figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,604 INFO [make_plots] added Port Renfrew figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,605 INFO [make_plots] added Sand Heads figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,605 INFO [make_plots] added Sandy Cove figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,606 INFO [make_plots] added Squamish figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,607 INFO [make_plots] added Victoria figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,608 INFO [make_plots] added Woodwards Landing figure to 2025-07-06 NEMO forecast2 publish render list
2025-07-07 03:48:38,609 INFO [make_plots] starting to render nemo forecast2 publish 06jul25 figures
2025-07-07 03:53:03,777 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 207, 38.3% complete
2025-07-07 03:56:20,032 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-07-07 03:56:20,033 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-07-07 03:56:45,238 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 03:57:50,565 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-07-07 03:57:50,566 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-07-07 03:58:03,799 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 309, 57.2% complete
2025-07-07 03:58:14,239 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 03:58:57,597 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 03:59:20,572 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 03:59:43,290 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 03:59:58,228 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/06jul25
2025-07-07 04:00:00,796 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 04:00:17,003 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 04:00:33,954 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 04:00:50,768 WARNING [make_plots] invalid station number for Squamish station: None
2025-07-07 04:00:50,769 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-07-07 04:01:07,375 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 04:01:23,824 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-07-02 00:00:00Z to 2025-07-09 05:55:00Z
2025-07-07 04:01:41,067 INFO [make_plots] finished rendering nemo forecast2 publish 06jul25 figures
2025-07-07 04:01:41,071 INFO [make_plots] nemo publish plots for 2025-07-06 forecast2 completed
2025-07-07 04:01:41,079 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-07-07 04:01:41,435 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast2', '--run-date', '2025-07-06'], host='localhost')
2025-07-07 04:01:44,819 INFO [make_feeds] running in process 1390167
2025-07-07 04:01:44,820 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 04:01:44,821 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-07-07 04:01:44,823 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-07-07 04:01:59,368 INFO [make_feeds] ATOM feeds for 2025-07-06 forecast2 run completed
2025-07-07 04:01:59,371 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-07-07 04:01:59,736 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.clear_checklist', args=[], host='localhost')
2025-07-07 04:02:00,800 INFO [clear_checklist] running in process 1390185
2025-07-07 04:02:00,801 INFO [clear_checklist] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 04:02:00,802 INFO [clear_checklist] publishing log messages to tcp://*:5564
2025-07-07 04:02:00,803 INFO [clear_checklist] connected to skookum.eos.ubc.ca port 5555
2025-07-07 04:02:00,804 INFO [clear_checklist] requesting that manager clear system state checklist
2025-07-07 04:02:00,807 INFO [manager] writing checklist to log file
2025-07-07 04:02:00,853 INFO [manager] checklist cleared
2025-07-07 04:02:00,855 INFO [clear_checklist] nowcast system checklist cleared
2025-07-07 04:02:00,858 INFO [manager] launching NextWorker(module='nemo_nowcast.workers.rotate_logs', args=[], host='localhost')
2025-07-07 04:02:01,998 INFO [rotate_logs] running in process 1390190
2025-07-07 04:02:02,000 INFO [rotate_logs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-07-07 04:02:02,001 INFO [rotate_logs] publishing log messages to tcp://*:5564
2025-07-07 04:02:02,002 INFO [rotate_logs] connected to skookum.eos.ubc.ca port 5555
2025-07-07 04:02:02,003 INFO [rotate_logs] rotating log files