2025-12-19 02:49:34,532 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-12-19 02:49:34,538 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-12-19 02:49:34,543 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-12-19 02:49:34,544 INFO [rotate_logs] log files rotated
2025-12-19 02:49:34,547 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-12-19 02:50:49,682 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/18dec25
2025-12-19 02:50:49,705 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-12-18 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/18dec25
2025-12-19 02:50:49,706 INFO [make_surface_current_tiles] surface current tile figures for 2025-12-18 forecast2 completed
2025-12-19 02:50:49,991 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-12-19 02:53:01,889 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 333, 61.7% complete
2025-12-19 02:58:01,889 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 414, 76.7% complete
2025-12-19 03:03:01,890 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 498, 92.2% complete
2025-12-19 03:08:01,891 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-19 03:08:01,895 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-19 03:08:02,222 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 03:08:03,680 INFO [download_wwatch3_results] running in process 1528685
2025-12-19 03:08:03,681 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 03:08:03,683 INFO [download_wwatch3_results] publishing log messages to tcp://*:5563
2025-12-19 03:08:03,684 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-19 03:08:40,394 INFO [download_wwatch3_results] forecast2 2025-12-19 results files from arbutus.cloud-nowcast downloaded
2025-12-19 03:08:40,398 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-19 03:08:40,545 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 03:08:41,744 INFO [update_forecast_datasets] running in process 1528701
2025-12-19 03:08:41,745 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 03:08:41,746 INFO [update_forecast_datasets] publishing log messages to tcp://*:5563
2025-12-19 03:08:41,748 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-19 03:08:41,749 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/19dec25 run
2025-12-19 03:08:41,752 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/19dec25 run
2025-12-19 03:08:41,753 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/19dec25 run
2025-12-19 03:09:05,052 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/19dec25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-12-19 03:09:05,053 INFO [update_forecast_datasets] wwatch3 2025-12-19 forecast2 rolling forecast datasets updated
2025-12-19 03:09:05,057 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-19 03:09:05,177 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-12-19 03:09:06,456 INFO [ping_erddap] running in process 1528713
2025-12-19 03:09:06,457 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 03:09:06,458 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-12-19 03:09:06,460 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-19 03:09:06,461 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-12-19 03:09:06,463 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-19 03:09:06,583 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 03:09:09,959 INFO [make_plots] running in process 1528718
2025-12-19 03:09:09,960 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 03:09:09,962 INFO [make_plots] publishing log messages to tcp://*:5563
2025-12-19 03:09:09,963 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-19 03:09:09,964 INFO [make_plots] preparing render list for 2025-12-19 WaveWatch3 forecast2 publish figures
2025-12-19 03:09:09,965 INFO [make_plots] added Halibut Bank figure to 2025-12-19 WaveWatch3 forecast2 publish render list
2025-12-19 03:09:09,966 INFO [make_plots] added Sentry Shoal figure to 2025-12-19 WaveWatch3 forecast2 publish render list
2025-12-19 03:09:09,967 INFO [make_plots] starting to render wwatch3 forecast2 publish 19dec25 figures
2025-12-19 03:09:10,590 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-12-19 03:09:16,663 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-12-19 03:09:20,865 INFO [make_plots] finished rendering wwatch3 forecast2 publish 19dec25 figures
2025-12-19 03:09:20,866 INFO [make_plots] wwatch3 publish plots for 2025-12-19 forecast2 completed
2025-12-19 03:09:20,870 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-19 07:45:06,006 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20251219/12/
2025-12-19 07:45:06,007 INFO [collect_weather] 2025-12-19 2.5km weather forecast 12 collection complete
2025-12-19 07:45:06,010 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-19 07:45:06,332 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-12-18'], host='localhost')
2025-12-19 07:45:06,339 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-12-18'], host='localhost')
2025-12-19 07:45:06,344 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-12-18'], host='localhost')
2025-12-19 07:45:06,348 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-12-18'], host='localhost')
2025-12-19 07:45:06,352 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-12-19 07:45:06,357 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-12-19 07:45:06,363 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-12-19 07:45:06,367 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-12-19 07:45:06,372 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-12-19 07:45:07,999 INFO [collect_NeahBay_ssh] running in process 1532093
2025-12-19 07:45:08,001 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:08,001 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5562
2025-12-19 07:45:08,006 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:08,007 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-12-19
2025-12-19 07:45:08,010 INFO [collect_weather] running in process 1532095
2025-12-19 07:45:08,011 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:08,012 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-12-19 07:45:08,012 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:08,025 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-12-19 07:45:08,111 INFO [download_live_ocean] running in process 1532094
2025-12-19 07:45:08,112 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:08,112 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-12-19 07:45:08,113 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:08,114 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-12-19
2025-12-19 07:45:08,370 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251219/12/
2025-12-19 07:45:08,371 INFO [crop_gribs] 2025-12-19 12 GRIBs cropping complete
2025-12-19 07:45:08,373 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-19 07:45:08,539 INFO [collect_river_data] running in process 1532089
2025-12-19 07:45:08,555 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:08,564 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-12-19 07:45:08,569 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:08,578 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-12-18
2025-12-19 07:45:08,579 INFO [collect_river_data] running in process 1532087
2025-12-19 07:45:08,580 INFO [download_live_ocean] downloaded 10146768 bytes from https://s3.kopah.uw.edu/liveocean-share/f2025.12.19/ubc.nc to /results/forcing/LiveOcean/downloaded/20251219/low_passed_UBC.nc
2025-12-19 07:45:08,581 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:08,582 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-12-19 07:45:08,583 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-12-19 07:45:08,584 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:08,585 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-12-18
2025-12-19 07:45:08,596 INFO [make_turbidity_file] running in process 1532092
2025-12-19 07:45:08,598 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:08,599 INFO [make_turbidity_file] publishing log messages to tcp://*:5567
2025-12-19 07:45:08,600 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:08,600 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-12-19
2025-12-19 07:45:08,639 INFO [collect_river_data] running in process 1532091
2025-12-19 07:45:08,640 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:08,641 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-12-19 07:45:08,643 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:08,644 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-12-18
2025-12-19 07:45:08,661 INFO [collect_river_data] running in process 1532090
2025-12-19 07:45:08,662 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:08,663 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-12-19 07:45:08,664 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:08,665 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-12-18
2025-12-19 07:45:08,694 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-12-19 07:45:09,024 INFO [crop_gribs] running in process 1532096
2025-12-19 07:45:09,030 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:09,031 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-12-19 07:45:09,032 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:09,033 INFO [crop_gribs] cropping 2025-12-19 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-12-19 07:45:09,039 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-12-18&endDT=2025-12-18 "HTTP/1.1 200 "
2025-12-19 07:45:09,043 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251219/18/
2025-12-19 07:45:09,044 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-12-18&endDT=2025-12-18 "HTTP/1.1 200 "
2025-12-19 07:45:09,045 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-12-18&endDT=2025-12-18 "HTTP/1.1 200 "
2025-12-19 07:45:09,046 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-12-18&endDT=2025-12-18 "HTTP/1.1 200 "
2025-12-19 07:45:09,054 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-12-18 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-12-19 07:45:09,055 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-12-18 completed
2025-12-19 07:45:09,056 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-19 07:45:09,083 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-12-18 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-12-19 07:45:09,084 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-12-18 completed
2025-12-19 07:45:09,142 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-12-18 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-12-19 07:45:09,143 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-12-18 completed
2025-12-19 07:45:09,151 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-12-18 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-12-19 07:45:09,153 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-12-18 completed
2025-12-19 07:45:09,183 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-19 07:45:09,327 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-19 07:45:09,464 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-12-19 07:45:09,602 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20251219/low_passed_UBC.nc
2025-12-19 07:45:09,604 INFO [download_live_ocean] 2025-12-19 Live Ocean file for Salish Sea western boundary downloaded
2025-12-19 07:45:09,678 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-12-19 07:45:09,992 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-12-19'], host='localhost')
2025-12-19 07:45:10,010 INFO [collect_NeahBay_ssh] 2025-12-19T00:00:00-08:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-12-19 07:45:10,012 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-12-19 07:45:10,179 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 07:45:10,712 INFO [grib_to_netcdf] running in process 1532205
2025-12-19 07:45:10,713 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:10,714 INFO [grib_to_netcdf] publishing log messages to tcp://*:5562
2025-12-19 07:45:10,715 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:10,716 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-12-19 nowcast and forecast runs
2025-12-19 07:45:11,340 INFO [make_turbidity_file] 2025-12-19 Fraser River turbidity file creation complete
2025-12-19 07:45:11,342 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-12-19 07:45:12,218 INFO [make_live_ocean_files] running in process 1532230
2025-12-19 07:45:12,219 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:12,220 INFO [make_live_ocean_files] publishing log messages to tcp://*:5564
2025-12-19 07:45:12,221 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:12,221 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-12-19 Live Ocean run
2025-12-19 07:45:12,888 INFO [make_ssh_files] running in process 1532232
2025-12-19 07:45:12,889 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:12,890 INFO [make_ssh_files] publishing log messages to tcp://*:5565
2025-12-19 07:45:12,892 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:12,893 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-12-19 NOAA Neah Bay 06Z observation and forecast values
2025-12-19 07:45:14,556 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d15.nc
2025-12-19 07:45:14,845 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d16.nc
2025-12-19 07:45:15,028 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d17.nc
2025-12-19 07:45:15,207 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m12d18.nc
2025-12-19 07:45:15,401 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d19.nc
2025-12-19 07:45:15,585 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d20.nc
2025-12-19 07:45:15,774 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d21.nc
2025-12-19 07:45:15,975 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m12d22.nc
2025-12-19 07:45:16,096 INFO [make_ssh_files] sea surface height boundary file for 2025-12-19 nowcast run created
2025-12-19 07:45:16,100 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-12-19 07:45:16,252 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-12-19 07:45:16,257 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-12-19 07:45:17,525 INFO [make_201702_runoff_file] running in process 1532316
2025-12-19 07:45:17,527 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:17,528 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5565
2025-12-19 07:45:17,529 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:17,565 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-12-19 07:45:17,671 INFO [make_201702_runoff_file] 2025-12-19 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-12-19 07:45:17,674 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-12-19 07:45:18,080 INFO [make_runoff_file] running in process 1532314
2025-12-19 07:45:18,081 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:18,082 INFO [make_runoff_file] publishing log messages to tcp://*:5565
2025-12-19 07:45:18,084 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:18,085 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-12-18
2025-12-19 07:45:20,907 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-12-18: /results/forcing/rivers/R202108Dailies_y2025m12d18.nc
2025-12-19 07:45:20,908 INFO [make_runoff_file] 2025-12-18 runoff file creation completed
2025-12-19 07:45:20,910 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-12-19 07:45:22,446 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m12d19.nc
2025-12-19 07:45:31,358 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d20.nc
2025-12-19 07:45:36,490 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m12d21.nc
2025-12-19 07:45:36,510 INFO [grib_to_netcdf] 2025-12-19 NEMO-atmos forcing file for nowcast+ created
2025-12-19 07:45:36,513 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-12-19 07:45:53,851 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m12d19.nc
2025-12-19 07:45:53,852 INFO [make_live_ocean_files] 2025-12-19 Live Ocean western boundary conditions files created
2025-12-19 07:45:53,855 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-12-19 07:45:54,011 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-12-19 07:45:54,016 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-12-19 07:45:54,022 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-12-19 07:45:54,027 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'nowcast+'], host='localhost')
2025-12-19 07:45:54,032 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-12-19 07:45:55,114 INFO [ping_erddap] running in process 1532395
2025-12-19 07:45:55,116 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:55,117 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-19 07:45:55,118 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:55,120 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-12-19 07:45:55,122 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-19 07:45:55,226 INFO [upload_forcing] running in process 1532398
2025-12-19 07:45:55,228 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:55,229 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-12-19 07:45:55,230 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:55,256 INFO [upload_forcing] running in process 1532399
2025-12-19 07:45:55,257 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:55,258 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-12-19 07:45:55,259 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:55,262 INFO [upload_forcing] running in process 1532400
2025-12-19 07:45:55,263 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:55,264 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-12-19 07:45:55,265 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:55,298 INFO [upload_forcing] running in process 1532397
2025-12-19 07:45:55,299 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:45:55,300 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-12-19 07:45:55,301 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:45:56,986 INFO [upload_forcing] nowcast+ 2025-12-19 forcing files upload to optimum-hindcast completed
2025-12-19 07:45:56,990 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-19 07:46:00,995 INFO [upload_forcing] nowcast+ 2025-12-19 forcing files upload to arbutus.cloud-nowcast completed
2025-12-19 07:46:00,999 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-19 07:46:01,150 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 07:46:02,513 INFO [upload_forcing] nowcast+ 2025-12-19 forcing files upload to orcinus-nowcast-agrif completed
2025-12-19 07:46:02,516 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-19 07:46:02,587 INFO [make_forcing_links] running in process 1532431
2025-12-19 07:46:02,588 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:46:02,589 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-12-19 07:46:02,590 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:46:03,747 INFO [make_forcing_links] nowcast+ 2025-12-19 forcing file links on arbutus.cloud-nowcast created
2025-12-19 07:46:03,751 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-19 07:46:03,905 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-19'], host='arbutus.cloud-nowcast')
2025-12-19 07:46:05,813 INFO [upload_forcing] nowcast+ 2025-12-19 forcing files upload to robot.nibi completed
2025-12-19 07:46:05,818 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-19 07:46:05,976 INFO [run_NEMO] running in process 31632
2025-12-19 07:46:05,977 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:46:05,978 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-19 07:46:05,989 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:46:06,682 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-19 07:46:06,990 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/19dec25nowcast_2025-12-19T074606.025357-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-19 07:46:07,003 INFO [run_NEMO] nowcast NEMO run for 2025-12-19 on arbutus.cloud-nowcast started
2025-12-19 07:46:07,007 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-19 07:46:07,159 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-12-19 07:46:09,154 INFO [watch_NEMO] running in process 31905
2025-12-19 07:46:09,156 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 07:46:09,157 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-19 07:46:09,158 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-19 07:46:09,224 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-19 07:51:09,226 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18913424 = 2025-12-19 05:08:40 UTC, 21.4% complete
2025-12-19 07:56:09,227 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18913911 = 2025-12-19 10:33:20 UTC, 44.0% complete
2025-12-19 08:01:09,229 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18914415 = 2025-12-19 16:09:20 UTC, 67.3% complete
2025-12-19 08:06:09,230 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18914933 = 2025-12-19 21:54:40 UTC, 91.3% complete
2025-12-19 08:11:09,234 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-12-19 08:11:09,238 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-19 08:11:09,564 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:11:09,570 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:11:10,695 INFO [download_results] running in process 1532744
2025-12-19 08:11:10,696 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:11:10,697 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-19 08:11:10,699 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:11:10,700 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/19dec25 to /results/SalishSea/nowcast-blue.202111
2025-12-19 08:11:10,832 INFO [make_forcing_links] running in process 1532743
2025-12-19 08:11:10,833 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:11:10,834 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-12-19 08:11:10,835 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:11:11,720 INFO [make_forcing_links] ssh 2025-12-19 forcing file links on arbutus.cloud-nowcast created
2025-12-19 08:11:11,724 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-19 08:11:11,914 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-19'], host='arbutus.cloud-nowcast')
2025-12-19 08:11:14,084 INFO [run_NEMO] running in process 32302
2025-12-19 08:11:14,086 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:11:14,087 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-19 08:11:14,089 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:11:14,871 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-19 08:11:15,188 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/19dec25forecast_2025-12-19T081114.152902-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-19 08:11:15,200 INFO [run_NEMO] forecast NEMO run for 2025-12-19 on arbutus.cloud-nowcast started
2025-12-19 08:11:15,218 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-19 08:11:15,372 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-12-19 08:11:17,409 INFO [watch_NEMO] running in process 32575
2025-12-19 08:11:17,410 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:11:17,412 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-19 08:11:17,413 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:11:17,510 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-19 08:12:11,434 INFO [download_results] nowcast 2025-12-19 results files from arbutus.cloud-nowcast downloaded
2025-12-19 08:12:11,439 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-19 08:12:11,590 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:12:11,595 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-12-18'], host='localhost')
2025-12-19 08:12:11,601 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:12:13,607 INFO [make_CHS_currents_file] running in process 1532768
2025-12-19 08:12:13,608 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:12:13,609 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-12-19 08:12:13,610 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:12:14,501 INFO [make_plots] running in process 1532767
2025-12-19 08:12:14,503 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:12:14,504 INFO [make_plots] publishing log messages to tcp://*:5564
2025-12-19 08:12:14,505 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:12:14,522 INFO [make_plots] running in process 1532765
2025-12-19 08:12:14,523 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:12:14,524 INFO [make_plots] publishing log messages to tcp://*:5565
2025-12-19 08:12:14,524 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:12:14,917 INFO [make_plots] preparing render list for 2025-12-18 NEMO nowcast-blue comparison figures
2025-12-19 08:12:14,947 INFO [make_plots] preparing render list for 2025-12-19 NEMO nowcast-blue research figures
2025-12-19 08:12:14,963 INFO [make_plots] added SH_wind figure to 2025-12-18 NEMO nowcast-blue comparison render list
2025-12-19 08:12:14,964 INFO [make_plots] added Compare_VENUS_East figure to 2025-12-18 NEMO nowcast-blue comparison render list
2025-12-19 08:12:14,965 INFO [make_plots] added Compare_VENUS_Central figure to 2025-12-18 NEMO nowcast-blue comparison render list
2025-12-19 08:12:14,966 INFO [make_plots] starting to render nemo nowcast comparison 18dec25 figures
2025-12-19 08:12:16,528 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-blue research render list
2025-12-19 08:12:17,535 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-blue research render list
2025-12-19 08:12:17,536 INFO [make_plots] added Currents_sections_and_surface figure to 2025-12-19 NEMO nowcast-blue research render list
2025-12-19 08:12:17,537 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-12-19 NEMO nowcast-blue research render list
2025-12-19 08:12:17,537 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-12-19 NEMO nowcast-blue research render list
2025-12-19 08:12:17,538 INFO [make_plots] starting to render nemo nowcast research 19dec25 figures
2025-12-19 08:12:48,015 INFO [make_CHS_currents_file] Made CHS currents file for 2025-12-19 for nowcast
2025-12-19 08:12:48,018 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-12-19 08:13:19,869 INFO [make_plots] finished rendering nemo nowcast research 19dec25 figures
2025-12-19 08:13:19,871 INFO [make_plots] nemo research plots for 2025-12-19 nowcast completed
2025-12-19 08:13:19,890 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-19 08:13:41,321 INFO [make_plots] finished rendering nemo nowcast comparison 18dec25 figures
2025-12-19 08:13:41,324 INFO [make_plots] nemo comparison plots for 2025-12-18 nowcast completed
2025-12-19 08:13:41,328 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-19 08:16:17,510 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18915569 = 2025-12-20 04:58:40 UTC, 13.8% complete
2025-12-19 08:21:17,512 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18916034 = 2025-12-20 10:08:40 UTC, 28.2% complete
2025-12-19 08:26:17,513 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18916486 = 2025-12-20 15:10:00 UTC, 42.1% complete
2025-12-19 08:31:17,514 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18916953 = 2025-12-20 20:21:20 UTC, 56.6% complete
2025-12-19 08:36:17,516 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18917430 = 2025-12-21 01:39:20 UTC, 71.3% complete
2025-12-19 08:41:17,517 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18917902 = 2025-12-21 06:54:00 UTC, 85.9% complete
2025-12-19 08:46:17,518 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18918360 = 2025-12-21 11:59:20 UTC, 100.0% complete
2025-12-19 08:51:17,522 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-12-19 08:51:17,526 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-19 08:51:17,856 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-19'], host='arbutus.cloud-nowcast')
2025-12-19 08:51:17,867 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-19'], host='arbutus.cloud-nowcast')
2025-12-19 08:51:17,871 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:51:17,876 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:51:17,882 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.nibi', 'turbidity', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:51:17,886 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:51:17,894 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:51:19,077 INFO [download_results] running in process 1533450
2025-12-19 08:51:19,078 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:19,079 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-19 08:51:19,081 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:19,082 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/19dec25 to /results/SalishSea/forecast.202111
2025-12-19 08:51:19,135 INFO [upload_forcing] running in process 1533449
2025-12-19 08:51:19,137 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:19,138 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-12-19 08:51:19,139 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:19,141 INFO [upload_forcing] running in process 1533448
2025-12-19 08:51:19,142 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:19,143 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-12-19 08:51:19,144 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:19,174 INFO [upload_forcing] running in process 1533446
2025-12-19 08:51:19,175 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:19,176 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-12-19 08:51:19,177 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:19,180 INFO [upload_forcing] running in process 1533445
2025-12-19 08:51:19,181 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:19,182 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-12-19 08:51:19,183 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:19,805 INFO [upload_forcing] turbidity 2025-12-19 forcing files upload to optimum-hindcast completed
2025-12-19 08:51:19,808 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-19 08:51:20,194 INFO [upload_forcing] turbidity 2025-12-19 forcing files upload to arbutus.cloud-nowcast completed
2025-12-19 08:51:20,199 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-19 08:51:20,363 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:51:20,801 INFO [make_ww3_wind_file] running in process 834
2025-12-19 08:51:20,803 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:20,804 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-12-19 08:51:20,814 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:20,815 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-12-19 forecast run
2025-12-19 08:51:20,907 INFO [make_ww3_current_file] running in process 832
2025-12-19 08:51:20,908 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:20,909 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-12-19 08:51:20,911 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:20,912 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-12-19 forecast run
2025-12-19 08:51:21,762 INFO [make_forcing_links] running in process 1533480
2025-12-19 08:51:21,764 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:21,765 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-12-19 08:51:21,767 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:22,930 INFO [make_forcing_links] nowcast-green 2025-12-19 forcing file links on arbutus.cloud-nowcast created
2025-12-19 08:51:22,935 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-19 08:51:23,120 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-19'], host='arbutus.cloud-nowcast')
2025-12-19 08:51:23,703 INFO [upload_forcing] turbidity 2025-12-19 forcing files upload to robot.nibi completed
2025-12-19 08:51:23,708 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-19 08:51:25,387 INFO [run_NEMO] running in process 1181
2025-12-19 08:51:25,388 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:25,391 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-12-19 08:51:25,392 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:25,436 INFO [upload_forcing] turbidity 2025-12-19 forcing files upload to orcinus-nowcast-agrif completed
2025-12-19 08:51:25,441 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-12-19 08:51:25,615 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:51:27,101 INFO [make_forcing_links] running in process 1533490
2025-12-19 08:51:27,103 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:27,106 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-12-19 08:51:27,107 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:27,868 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-12-19 08:51:28,694 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/19dec25nowcast-green_2025-12-19T085125.456392-0800/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-12-19 08:51:28,747 INFO [run_NEMO] nowcast-green NEMO run for 2025-12-19 on arbutus.cloud-nowcast started
2025-12-19 08:51:28,751 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-12-19 08:51:28,911 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-12-19 08:51:32,156 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-12-19 forecast run
2025-12-19 08:51:32,160 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-12-19 08:51:32,729 INFO [watch_NEMO] running in process 1793
2025-12-19 08:51:32,731 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:32,732 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-12-19 08:51:32,733 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:32,797 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-12-19 08:51:33,025 INFO [make_forcing_links] nowcast-agrif 2025-12-19 forcing file links on orcinus-nowcast-agrif created
2025-12-19 08:51:33,030 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-12-19 08:51:33,193 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:51:34,724 INFO [run_NEMO_agrif] running in process 1533503
2025-12-19 08:51:34,726 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:51:34,727 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5564
2025-12-19 08:51:34,729 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:51:40,536 CRITICAL [run_NEMO_agrif] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 109, in run_NEMO_agrif
    prev_run_namelists_info = _get_prev_run_namelists_info(
        sftp_client, host_name, run_date.shift(days=-1), config
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/run_NEMO_agrif.py", line 155, in _get_prev_run_namelists_info
    sftp_client.get(os.fspath(prev_namelist_cfg), namelist_cfg.name)
    ~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 840, in get
    size = self.getfo(
        remotepath,
    ...<3 lines>...
        max_concurrent_prefetch_requests,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 795, in getfo
    file_size = self.stat(remotepath).st_size
                ~~~~~~~~~^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 493, in stat
    t, msg = self._request(CMD_STAT, path)
             ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 857, in _request
    return self._read_response(num)
           ~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 909, in _read_response
    self._convert_status(msg)
    ~~~~~~~~~~~~~~~~~~~~^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/paramiko/sftp_client.py", line 938, in _convert_status
    raise IOError(errno.ENOENT, text)
FileNotFoundError: [Errno 2] No such file
2025-12-19 08:52:24,798 INFO [download_results] forecast 2025-12-19 results files from arbutus.cloud-nowcast downloaded
2025-12-19 08:52:24,803 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-19 08:52:24,962 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:52:27,413 INFO [make_CHS_currents_file] running in process 1533526
2025-12-19 08:52:27,414 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:52:27,416 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-12-19 08:52:27,417 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:53:14,221 INFO [make_CHS_currents_file] Made CHS currents file for 2025-12-19 for forecast
2025-12-19 08:53:14,224 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-12-19 08:53:14,381 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:53:15,360 INFO [update_forecast_datasets] running in process 1533563
2025-12-19 08:53:15,361 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:53:15,361 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-12-19 08:53:15,362 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:53:15,363 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/19dec25 run
2025-12-19 08:53:15,819 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/19dec25 run: /results/SalishSea/rolling-forecasts/nemo
2025-12-19 08:53:15,820 INFO [update_forecast_datasets] nemo 2025-12-19 forecast rolling forecast datasets updated
2025-12-19 08:53:15,823 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-19 08:53:15,984 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-12-19 08:53:15,989 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:53:15,993 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 08:53:17,047 INFO [ping_erddap] running in process 1533568
2025-12-19 08:53:17,048 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:53:17,048 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-19 08:53:17,049 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:53:17,061 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-12-19 08:53:17,063 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-19 08:53:18,166 INFO [make_surface_current_tiles] running in process 1533571
2025-12-19 08:53:18,167 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:53:18,168 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5562
2025-12-19 08:53:18,169 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:53:18,187 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-12-19 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/19dec25
2025-12-19 08:53:19,394 INFO [make_plots] running in process 1533570
2025-12-19 08:53:19,396 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:53:19,397 INFO [make_plots] publishing log messages to tcp://*:5564
2025-12-19 08:53:19,399 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:53:19,933 INFO [make_plots] preparing render list for 2025-12-19 NEMO forecast publish figures
2025-12-19 08:53:20,847 INFO [make_plots] added Website_thumbnail figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,848 INFO [make_plots] added Threshold_website figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,848 INFO [make_plots] added PA_tidal_predictions figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,849 INFO [make_plots] added Boundary Bay figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,849 INFO [make_plots] added Campbell River figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,850 INFO [make_plots] added Cherry Point figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,851 INFO [make_plots] added Friday Harbor figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,851 INFO [make_plots] added Halfmoon Bay figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,852 INFO [make_plots] added Nanaimo figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,853 INFO [make_plots] added Neah Bay figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,853 INFO [make_plots] added New Westminster figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,854 INFO [make_plots] added Patricia Bay figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,855 INFO [make_plots] added Point Atkinson figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,856 INFO [make_plots] added Port Renfrew figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,856 INFO [make_plots] added Sand Heads figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,857 INFO [make_plots] added Sandy Cove figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,858 INFO [make_plots] added Squamish figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,858 INFO [make_plots] added Victoria figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,859 INFO [make_plots] added Woodwards Landing figure to 2025-12-19 NEMO forecast publish render list
2025-12-19 08:53:20,859 INFO [make_plots] starting to render nemo forecast publish 19dec25 figures
2025-12-19 08:53:57,695 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-12-19 forecast run
2025-12-19 08:53:57,699 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-12-19 08:53:58,011 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-19'], host='arbutus.cloud-nowcast')
2025-12-19 08:54:00,017 INFO [run_ww3] running in process 1901
2025-12-19 08:54:00,018 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:54:00,019 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-12-19 08:54:00,021 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:54:00,024 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-12-19T085400.016190-0800
2025-12-19 08:54:00,025 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-12-19T085400.016190-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-12-19 08:54:00,045 INFO [run_ww3] nowcast WaveWatch3 run for 2025-12-19 on arbutus.cloud-nowcast started
2025-12-19 08:54:00,063 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-12-19 08:54:00,370 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-12-19 08:54:02,337 INFO [watch_ww3] running in process 1988
2025-12-19 08:54:02,339 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 08:54:02,340 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-12-19 08:54:02,341 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-19 08:54:02,370 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-19 08:56:32,798 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18913119 = 2025-12-19 01:45:20 UTC, 7.3% complete
2025-12-19 08:59:02,369 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 75, 17.4% complete
2025-12-19 09:00:34,775 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-12-19 09:00:34,775 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-12-19 09:01:02,626 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:01:32,799 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18913296 = 2025-12-19 03:43:20 UTC, 15.5% complete
2025-12-19 09:02:20,959 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-12-19 09:02:20,959 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-12-19 09:02:45,604 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:03:31,364 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:03:38,845 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/19dec25
2025-12-19 09:03:51,637 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:04:02,373 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 165, 38.2% complete
2025-12-19 09:04:12,500 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:04:32,968 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:04:51,261 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:05:12,415 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:05:32,347 WARNING [make_plots] invalid station number for Squamish station: None
2025-12-19 09:05:32,347 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-12-19 09:05:52,486 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:06:11,182 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-12-14 00:00:00Z to 2025-12-21 11:55:00Z
2025-12-19 09:06:30,704 INFO [make_plots] finished rendering nemo forecast publish 19dec25 figures
2025-12-19 09:06:30,707 INFO [make_plots] nemo publish plots for 2025-12-19 forecast completed
2025-12-19 09:06:30,713 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-19 09:06:30,874 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 09:06:32,807 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18913473 = 2025-12-19 05:41:20 UTC, 23.7% complete
2025-12-19 09:06:34,214 INFO [make_feeds] running in process 1538257
2025-12-19 09:06:34,215 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 09:06:34,216 INFO [make_feeds] publishing log messages to tcp://*:5564
2025-12-19 09:06:34,218 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-12-19 09:06:48,523 INFO [make_feeds] ATOM feeds for 2025-12-19 forecast run completed
2025-12-19 09:06:48,526 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-12-19 09:08:21,387 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/19dec25
2025-12-19 09:08:21,411 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-12-19 forecast into /results/nowcast-sys/figures/surface_currents/forecast/19dec25
2025-12-19 09:08:21,413 INFO [make_surface_current_tiles] surface current tile figures for 2025-12-19 forecast completed
2025-12-19 09:08:21,722 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-12-19 09:09:02,371 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 261, 60.4% complete
2025-12-19 09:11:32,801 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18913650 = 2025-12-19 07:39:20 UTC, 31.9% complete
2025-12-19 09:14:02,371 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 354, 81.9% complete
2025-12-19 09:16:32,802 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18913828 = 2025-12-19 09:38:00 UTC, 40.2% complete
2025-12-19 09:19:02,372 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-19 09:21:32,802 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18914005 = 2025-12-19 11:36:00 UTC, 48.4% complete
2025-12-19 09:24:02,373 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-19 09:24:02,376 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-19 09:24:02,876 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 09:24:02,884 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-19'], host='arbutus.cloud-nowcast')
2025-12-19 09:24:04,203 INFO [download_wwatch3_results] running in process 1538502
2025-12-19 09:24:04,204 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 09:24:04,205 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-12-19 09:24:04,207 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-19 09:24:04,895 INFO [run_ww3] running in process 2251
2025-12-19 09:24:04,897 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 09:24:04,899 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-12-19 09:24:04,900 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-19 09:24:04,904 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-12-19T092404.890866-0800
2025-12-19 09:24:04,905 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-12-19T092404.890866-0800/SoGWW3.sh on arbutus.cloud-nowcast
2025-12-19 09:24:04,914 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-12-19 09:24:04,918 INFO [run_ww3] forecast WaveWatch3 run for 2025-12-19 on arbutus.cloud-nowcast started
2025-12-19 09:24:05,206 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-12-19 09:24:07,160 INFO [watch_ww3] running in process 2408
2025-12-19 09:24:07,162 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 09:24:07,163 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-12-19 09:24:07,164 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-12-19 09:24:07,197 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-19 09:24:39,004 INFO [download_wwatch3_results] nowcast 2025-12-19 results files from arbutus.cloud-nowcast downloaded
2025-12-19 09:24:39,008 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-19 09:26:32,803 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18914181 = 2025-12-19 13:33:20 UTC, 56.5% complete
2025-12-19 09:29:07,199 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 72, 11.1% complete
2025-12-19 09:31:32,804 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18914359 = 2025-12-19 15:32:00 UTC, 64.8% complete
2025-12-19 09:34:07,198 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 153, 23.6% complete
2025-12-19 09:36:32,806 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18914535 = 2025-12-19 17:29:20 UTC, 72.9% complete
2025-12-19 09:39:07,199 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 234, 36.1% complete
2025-12-19 09:41:32,808 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18914712 = 2025-12-19 19:27:20 UTC, 81.1% complete
2025-12-19 09:44:07,201 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 318, 49.1% complete
2025-12-19 09:46:32,809 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18914889 = 2025-12-19 21:25:20 UTC, 89.3% complete
2025-12-19 09:49:07,202 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 408, 63.0% complete
2025-12-19 09:51:32,810 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18915066 = 2025-12-19 23:23:20 UTC, 97.5% complete
2025-12-19 09:54:07,203 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 498, 76.9% complete
2025-12-19 09:56:32,812 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18915120 = 2025-12-19 23:59:20 UTC, 100.0% complete
2025-12-19 09:59:07,208 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 588, 90.7% complete
2025-12-19 10:01:32,819 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-12-19 10:01:32,825 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-12-19 10:01:33,299 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 10:01:34,475 INFO [download_results] running in process 1538962
2025-12-19 10:01:34,476 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:01:34,477 INFO [download_results] publishing log messages to tcp://*:5562
2025-12-19 10:01:34,478 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:01:34,479 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/19dec25 to /results2/SalishSea/nowcast-green.202111
2025-12-19 10:04:07,206 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-12-19 10:05:01,151 INFO [download_results] nowcast-green 2025-12-19 results files from arbutus.cloud-nowcast downloaded
2025-12-19 10:05:01,157 INFO [manager] checklist updated with [results files] items from download_results worker
2025-12-19 10:05:01,481 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 10:05:01,487 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-12-19 10:05:01,494 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 10:05:01,498 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 10:05:01,504 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 10:05:02,761 INFO [ping_erddap] running in process 1539022
2025-12-19 10:05:02,762 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:05:02,763 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-19 10:05:02,764 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:05:02,768 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-12-19 10:05:02,770 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-19 10:05:03,878 INFO [make_averaged_dataset] running in process 1539025
2025-12-19 10:05:03,879 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:05:03,880 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2025-12-19 10:05:03,881 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:05:03,881 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Dec-2025 biology
2025-12-19 10:05:03,898 INFO [make_averaged_dataset] running in process 1539023
2025-12-19 10:05:03,899 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:05:03,900 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-12-19 10:05:03,900 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:05:03,901 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Dec-2025 physics
2025-12-19 10:05:03,904 INFO [make_averaged_dataset] running in process 1539024
2025-12-19 10:05:03,905 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:05:03,906 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-12-19 10:05:03,907 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:05:03,908 INFO [make_averaged_dataset] creating day-averaged dataset for 19-Dec-2025 chemistry
2025-12-19 10:05:04,820 INFO [make_plots] running in process 1539020
2025-12-19 10:05:04,821 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:05:04,822 INFO [make_plots] publishing log messages to tcp://*:5566
2025-12-19 10:05:04,823 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:05:05,243 INFO [make_plots] preparing render list for 2025-12-19 NEMO nowcast-green research figures
2025-12-19 10:05:07,098 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:07,872 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:08,681 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:09,439 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:10,267 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:11,127 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:11,887 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:12,768 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:13,636 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:14,528 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:15,414 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:16,171 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:17,294 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:17,294 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:17,296 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:17,296 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-12-19 NEMO nowcast-green research render list
2025-12-19 10:05:17,297 INFO [make_plots] starting to render nemo nowcast-green research 19dec25 figures
2025-12-19 10:05:37,408 INFO [make_averaged_dataset] day-averaged dataset for 19-Dec-2025 chemistry created
2025-12-19 10:05:37,411 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-19 10:06:00,436 INFO [make_averaged_dataset] day-averaged dataset for 19-Dec-2025 physics created
2025-12-19 10:06:00,439 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-19 10:06:16,276 INFO [make_averaged_dataset] day-averaged dataset for 19-Dec-2025 biology created
2025-12-19 10:06:16,279 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-12-19 10:09:07,207 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-12-19 10:09:07,210 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-12-19 10:09:07,687 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 10:09:08,909 INFO [download_wwatch3_results] running in process 1539236
2025-12-19 10:09:08,911 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:09:08,913 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-12-19 10:09:08,914 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:09:50,051 INFO [download_wwatch3_results] forecast 2025-12-19 results files from arbutus.cloud-nowcast downloaded
2025-12-19 10:09:50,055 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-12-19 10:09:50,381 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 10:09:51,505 INFO [update_forecast_datasets] running in process 1539248
2025-12-19 10:09:51,506 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:09:51,507 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-12-19 10:09:51,508 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:09:51,509 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/19dec25 run
2025-12-19 10:09:51,512 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/19dec25 run
2025-12-19 10:09:51,513 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/19dec25 run
2025-12-19 10:09:51,591 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/19dec25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-12-19 10:09:51,592 INFO [update_forecast_datasets] wwatch3 2025-12-19 forecast rolling forecast datasets updated
2025-12-19 10:09:51,595 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-12-19 10:09:51,897 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-12-19 10:09:52,920 INFO [ping_erddap] running in process 1539253
2025-12-19 10:09:52,921 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:09:52,922 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-12-19 10:09:52,923 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:09:52,925 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-12-19 10:09:52,927 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-12-19 10:09:53,217 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-12-19'], host='localhost')
2025-12-19 10:09:56,648 INFO [make_plots] running in process 1539258
2025-12-19 10:09:56,649 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 10:09:56,650 INFO [make_plots] publishing log messages to tcp://*:5562
2025-12-19 10:09:56,651 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-12-19 10:09:56,652 INFO [make_plots] preparing render list for 2025-12-19 WaveWatch3 forecast publish figures
2025-12-19 10:09:56,653 INFO [make_plots] added Halibut Bank figure to 2025-12-19 WaveWatch3 forecast publish render list
2025-12-19 10:09:56,654 INFO [make_plots] added Sentry Shoal figure to 2025-12-19 WaveWatch3 forecast publish render list
2025-12-19 10:09:56,655 INFO [make_plots] starting to render wwatch3 forecast publish 19dec25 figures
2025-12-19 10:09:57,328 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-12-19 10:10:01,501 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-12-19 10:10:05,674 INFO [make_plots] finished rendering wwatch3 forecast publish 19dec25 figures
2025-12-19 10:10:05,675 INFO [make_plots] wwatch3 publish plots for 2025-12-19 forecast completed
2025-12-19 10:10:05,679 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-19 11:02:10,634 INFO [make_plots] finished rendering nemo nowcast-green research 19dec25 figures
2025-12-19 11:02:10,663 INFO [make_plots] nemo research plots for 2025-12-19 nowcast-green completed
2025-12-19 11:02:10,729 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-12-19 13:51:16,706 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20251219/18/
2025-12-19 13:51:16,707 INFO [collect_weather] 2025-12-19 2.5km weather forecast 18 collection complete
2025-12-19 13:51:16,711 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-12-19 13:51:17,146 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20251219/18/
2025-12-19 13:51:17,147 INFO [crop_gribs] 2025-12-19 18 GRIBs cropping complete
2025-12-19 13:51:17,225 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-12-19 13:51:17,233 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-12-19 13:51:17,238 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-12-19 13:51:17,243 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-12-20'], host='localhost')
2025-12-19 13:51:17,250 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-12-19 13:51:18,943 INFO [download_weather] running in process 1541888
2025-12-19 13:51:18,944 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 13:51:18,945 INFO [collect_weather] running in process 1541889
2025-12-19 13:51:18,946 INFO [download_weather] publishing log messages to tcp://*:5562
2025-12-19 13:51:18,947 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 13:51:18,948 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-19 13:51:18,949 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-12-19 13:51:18,950 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20251219
2025-12-19 13:51:18,950 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-19 13:51:18,951 INFO [download_weather] running in process 1541886
2025-12-19 13:51:18,952 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 13:51:18,952 INFO [download_weather] publishing log messages to tcp://*:5564
2025-12-19 13:51:18,953 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-12-19 13:51:18,954 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20251219
2025-12-19 13:51:18,989 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-12-19 13:51:20,053 INFO [crop_gribs] running in process 1541890
2025-12-19 13:51:20,054 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-12-19 13:51:20,055 INFO [crop_gribs] publishing log messages to tcp://*:5565
2025-12-19 13:51:20,056 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-12-19 13:51:20,058 INFO [crop_gribs] cropping 2025-12-20 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-12-19 13:51:20,073 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20251220/00/
2025-12-19 13:54:56,246 INFO [download_weather] 2025-12-19 1km weather forecast 12 downloads complete
2025-12-19 13:54:56,249 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-12-19 13:55:04,071 INFO [download_weather] 2025-12-19 1km weather forecast 00 downloads complete
2025-12-19 13:55:04,073 INFO [manager] checklist updated with [weather forecast] items from download_weather worker