2025-03-21 04:30:05,693 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-03-21 04:30:05,698 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-03-21 04:30:05,702 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-03-21 04:30:05,702 INFO [rotate_logs] log files rotated
2025-03-21 04:30:05,709 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-03-21 04:47:25,546 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/20mar25
2025-03-21 04:56:24,375 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/20mar25
2025-03-21 04:56:24,420 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-03-20 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/20mar25
2025-03-21 04:56:24,421 INFO [make_surface_current_tiles] surface current tile figures for 2025-03-20 forecast2 completed
2025-03-21 04:56:24,997 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-03-21 08:59:03,461 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250321/12/
2025-03-21 08:59:03,489 INFO [collect_weather] 2025-03-21 2.5km weather forecast 12 collection complete
2025-03-21 08:59:03,567 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-03-21 08:59:04,438 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-03-20'], host='localhost')
2025-03-21 08:59:04,488 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250321/12/
2025-03-21 08:59:04,496 INFO [crop_gribs] 2025-03-21 12 GRIBs cropping complete
2025-03-21 08:59:04,516 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-03-20'], host='localhost')
2025-03-21 08:59:04,536 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-03-20'], host='localhost')
2025-03-21 08:59:04,564 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-03-20'], host='localhost')
2025-03-21 08:59:04,578 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-03-21 08:59:04,596 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-03-21 08:59:04,616 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-03-21 08:59:04,652 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-03-21 08:59:04,677 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-03-21 08:59:04,708 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-03-21 08:59:05,265 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-03-21 08:59:13,064 INFO [download_live_ocean] running in process 2441530
2025-03-21 08:59:13,066 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:13,067 INFO [download_live_ocean] publishing log messages to tcp://*:5562
2025-03-21 08:59:13,069 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:13,070 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-03-21
2025-03-21 08:59:13,176 INFO [collect_weather] running in process 2441531
2025-03-21 08:59:13,188 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:13,196 INFO [collect_weather] publishing log messages to tcp://*:5563
2025-03-21 08:59:13,197 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:13,198 INFO [collect_NeahBay_ssh] running in process 2441529
2025-03-21 08:59:13,199 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:13,201 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5564
2025-03-21 08:59:13,202 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:13,204 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-03-21
2025-03-21 08:59:13,368 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-03-21 08:59:13,712 INFO [download_live_ocean] downloaded 10180324 bytes from https://liveocean.apl.uw.edu/output/f2025.03.21/ubc.nc to /results/forcing/LiveOcean/downloaded/20250321/low_passed_UBC.nc
2025-03-21 08:59:13,720 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-03-21 08:59:14,877 INFO [collect_NeahBay_ssh] 2025-03-21T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-03-21 08:59:14,896 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-03-21 08:59:15,312 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 08:59:15,781 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250321/low_passed_UBC.nc
2025-03-21 08:59:15,782 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-03-21 08:59:15,784 INFO [download_live_ocean] 2025-03-21 Live Ocean file for Salish Sea western boundary downloaded
2025-03-21 08:59:16,004 INFO [collect_river_data] running in process 2441523
2025-03-21 08:59:16,007 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:16,008 INFO [collect_river_data] publishing log messages to tcp://*:5564
2025-03-21 08:59:16,011 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:16,013 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-03-20
2025-03-21 08:59:16,016 INFO [collect_river_data] running in process 2441525
2025-03-21 08:59:16,019 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:16,021 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-03-21 08:59:16,022 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:16,025 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-03-20
2025-03-21 08:59:16,026 INFO [collect_river_data] running in process 2441526
2025-03-21 08:59:16,030 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:16,040 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-03-21 08:59:16,062 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:16,080 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-03-20
2025-03-21 08:59:16,199 INFO [collect_river_data] running in process 2441527
2025-03-21 08:59:16,240 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:16,267 INFO [collect_river_data] publishing log messages to tcp://*:5568
2025-03-21 08:59:16,279 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:16,280 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-03-20
2025-03-21 08:59:16,393 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-03-21'], host='localhost')
2025-03-21 08:59:16,533 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-03-20&endDT=2025-03-20 "HTTP/1.1 200 OK"
2025-03-21 08:59:16,554 INFO [grib_to_netcdf] running in process 2441534
2025-03-21 08:59:16,561 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-03-20&endDT=2025-03-20 "HTTP/1.1 200 OK"
2025-03-21 08:59:16,564 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:16,568 INFO [grib_to_netcdf] publishing log messages to tcp://*:5569
2025-03-21 08:59:16,577 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:16,584 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-03-21 nowcast and forecast runs
2025-03-21 08:59:16,594 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-03-20&endDT=2025-03-20 "HTTP/1.1 200 OK"
2025-03-21 08:59:16,626 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-03-20&endDT=2025-03-20 "HTTP/1.1 200 OK"
2025-03-21 08:59:16,628 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-03-20 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-03-21 08:59:16,642 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-03-20 completed
2025-03-21 08:59:16,665 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-03-20 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-03-21 08:59:16,666 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-03-20 completed
2025-03-21 08:59:16,675 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-03-21 08:59:16,677 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-03-20 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-03-21 08:59:16,678 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-03-20 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-03-21 08:59:16,681 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-03-20 completed
2025-03-21 08:59:16,691 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-03-20 completed
2025-03-21 08:59:17,250 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-03-21 08:59:17,639 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-03-21 08:59:17,665 INFO [make_turbidity_file] running in process 2441528
2025-03-21 08:59:17,669 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:17,672 INFO [make_turbidity_file] publishing log messages to tcp://*:5562
2025-03-21 08:59:17,675 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:17,676 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-03-21
2025-03-21 08:59:17,976 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-03-21 08:59:19,008 INFO [crop_gribs] running in process 2441532
2025-03-21 08:59:19,040 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:19,076 INFO [crop_gribs] publishing log messages to tcp://*:5564
2025-03-21 08:59:19,096 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:19,108 INFO [crop_gribs] cropping 2025-03-21 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-03-21 08:59:19,148 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250321/18/
2025-03-21 08:59:22,787 INFO [make_turbidity_file] 2025-03-21 Fraser River turbidity file creation complete
2025-03-21 08:59:22,800 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-03-21 08:59:29,636 INFO [make_live_ocean_files] running in process 2441653
2025-03-21 08:59:29,671 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:29,725 INFO [make_live_ocean_files] publishing log messages to tcp://*:5562
2025-03-21 08:59:29,764 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:29,785 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-03-21 Live Ocean run
2025-03-21 08:59:29,786 INFO [make_ssh_files] running in process 2441639
2025-03-21 08:59:29,787 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:29,800 INFO [make_ssh_files] publishing log messages to tcp://*:5566
2025-03-21 08:59:29,818 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:29,829 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-03-21 NOAA Neah Bay 06Z observation and forecast values
2025-03-21 08:59:34,284 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m03d17.nc
2025-03-21 08:59:34,653 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m03d18.nc
2025-03-21 08:59:35,004 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m03d19.nc
2025-03-21 08:59:35,375 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m03d20.nc
2025-03-21 08:59:35,850 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m03d21.nc
2025-03-21 08:59:36,205 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m03d22.nc
2025-03-21 08:59:36,574 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m03d23.nc
2025-03-21 08:59:36,886 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m03d24.nc
2025-03-21 08:59:37,240 INFO [make_ssh_files] sea surface height boundary file for 2025-03-21 nowcast run created
2025-03-21 08:59:37,256 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-03-21 08:59:37,684 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=[], host='localhost')
2025-03-21 08:59:37,748 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-03-21 08:59:48,306 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m03d21.nc
2025-03-21 08:59:48,816 INFO [make_201702_runoff_file] running in process 2441713
2025-03-21 08:59:48,818 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:48,818 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5566
2025-03-21 08:59:48,820 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:48,993 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-03-21 08:59:49,231 INFO [make_201702_runoff_file] 2025-03-21 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-03-21 08:59:49,329 INFO [manager] checklist updated with [v201702 rivers forcing] items from make_201702_runoff_file worker
2025-03-21 08:59:52,792 INFO [make_runoff_file] running in process 2441711
2025-03-21 08:59:52,793 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 08:59:52,795 INFO [make_runoff_file] publishing log messages to tcp://*:5566
2025-03-21 08:59:52,796 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-03-21 08:59:52,797 INFO [make_runoff_file] calculating NEMO runoff forcing for 202108 bathymetry for 2025-03-20
2025-03-21 09:00:04,108 INFO [make_runoff_file] stored NEMO runoff forcing for 202108 bathymetry for 2025-03-20: /results/forcing/rivers/R202108Dailies_y2025m03d20.nc
2025-03-21 09:00:04,117 INFO [make_runoff_file] 2025-03-20 runoff file creation completed
2025-03-21 09:00:04,140 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-03-21 09:00:10,645 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m03d22.nc
2025-03-21 09:00:23,802 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m03d23.nc
2025-03-21 09:00:23,824 INFO [grib_to_netcdf] 2025-03-21 NEMO-atmos forcing file for nowcast+ created
2025-03-21 09:00:23,838 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-03-21 09:01:03,793 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m03d21.nc
2025-03-21 09:01:03,794 INFO [make_live_ocean_files] 2025-03-21 Live Ocean western boundary conditions files created
2025-03-21 09:01:03,796 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-03-21 09:01:04,259 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-03-21 09:01:04,272 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-03-21 09:01:04,300 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-03-21 09:01:04,320 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-03-21 09:01:04,356 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-03-21 09:01:11,145 INFO [ping_erddap] running in process 2441765
2025-03-21 09:01:11,145 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:01:11,189 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-03-21 09:01:11,193 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:01:11,196 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-03-21 09:01:11,199 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-03-21 09:01:11,538 INFO [upload_forcing] running in process 2441767
2025-03-21 09:01:11,539 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:01:11,539 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-03-21 09:01:11,540 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:01:11,544 INFO [upload_forcing] running in process 2441769
2025-03-21 09:01:11,545 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:01:11,546 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-03-21 09:01:11,554 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:01:11,596 INFO [upload_forcing] running in process 2441770
2025-03-21 09:01:11,600 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:01:11,601 INFO [upload_forcing] publishing log messages to tcp://*:5568
2025-03-21 09:01:11,602 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:01:11,604 INFO [upload_forcing] running in process 2441768
2025-03-21 09:01:11,607 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:01:11,612 INFO [upload_forcing] publishing log messages to tcp://*:5569
2025-03-21 09:01:11,620 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:01:15,354 INFO [upload_forcing] nowcast+ 2025-03-21 forcing files upload to optimum-hindcast completed
2025-03-21 09:01:15,369 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-03-21 09:01:15,396 INFO [upload_forcing] nowcast+ 2025-03-21 forcing files upload to orcinus-nowcast-agrif completed
2025-03-21 09:01:15,610 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-03-21 09:01:21,580 INFO [upload_forcing] nowcast+ 2025-03-21 forcing files upload to robot.graham completed
2025-03-21 09:01:21,584 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-03-21 09:01:26,459 INFO [upload_forcing] nowcast+ 2025-03-21 forcing files upload to arbutus.cloud-nowcast completed
2025-03-21 09:01:26,524 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-03-21 09:01:26,784 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 09:01:30,867 INFO [make_forcing_links] running in process 2441807
2025-03-21 09:01:30,873 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:01:30,877 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-03-21 09:01:30,881 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:01:32,143 INFO [make_forcing_links] nowcast+ 2025-03-21 forcing file links on arbutus.cloud-nowcast created
2025-03-21 09:01:32,157 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-03-21 09:01:32,522 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-03-21'], host='arbutus.cloud-nowcast')
2025-03-21 09:01:34,615 INFO [run_NEMO] running in process 10876
2025-03-21 09:01:34,616 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:01:34,617 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-03-21 09:01:34,740 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:01:36,009 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/21mar25nowcast_2025-03-21T090134.657948-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-03-21 09:01:36,074 INFO [run_NEMO] nowcast NEMO run for 2025-03-21 on arbutus.cloud-nowcast started
2025-03-21 09:01:36,109 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-03-21 09:01:36,256 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-03-21 09:01:38,173 INFO [watch_NEMO] running in process 11129
2025-03-21 09:01:38,203 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:01:38,207 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-03-21 09:01:38,208 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:01:38,233 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-03-21 09:06:38,234 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18323727 = 2025-03-21 04:57:20 UTC, 20.7% complete
2025-03-21 09:11:38,241 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18324194 = 2025-03-21 10:08:40 UTC, 42.3% complete
2025-03-21 09:16:38,237 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18324683 = 2025-03-21 15:34:40 UTC, 64.9% complete
2025-03-21 09:21:38,237 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18325159 = 2025-03-21 20:52:00 UTC, 87.0% complete
2025-03-21 09:26:38,240 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-03-21 09:26:38,265 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-03-21 09:26:38,831 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 09:26:38,842 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 09:26:42,312 INFO [download_results] running in process 2442105
2025-03-21 09:26:42,314 INFO [make_forcing_links] running in process 2442104
2025-03-21 09:26:42,316 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:26:42,317 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:26:42,318 INFO [download_results] publishing log messages to tcp://*:5562
2025-03-21 09:26:42,319 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2025-03-21 09:26:42,319 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:26:42,319 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:26:42,320 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/21mar25 to /results/SalishSea/nowcast-blue.202111
2025-03-21 09:26:43,333 INFO [make_forcing_links] ssh 2025-03-21 forcing file links on arbutus.cloud-nowcast created
2025-03-21 09:26:43,337 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-03-21 09:26:43,668 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-03-21'], host='arbutus.cloud-nowcast')
2025-03-21 09:26:50,642 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/21mar25forecast_2025-03-21T092649.218369-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-03-21 09:26:50,650 INFO [run_NEMO] forecast NEMO run for 2025-03-21 on arbutus.cloud-nowcast started
2025-03-21 09:26:50,657 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-03-21 09:26:50,955 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-03-21 09:27:02,470 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-03-21 09:32:02,278 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18325890 = 2025-03-22 04:59:20 UTC, 13.9% complete
2025-03-21 09:37:02,467 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18326380 = 2025-03-22 10:26:00 UTC, 29.0% complete
2025-03-21 09:42:02,472 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18326882 = 2025-03-22 16:00:40 UTC, 44.5% complete
2025-03-21 09:46:57,959 INFO [download_results] nowcast 2025-03-21 results files from arbutus.cloud-nowcast downloaded
2025-03-21 09:46:57,968 INFO [manager] checklist updated with [results files] items from download_results worker
2025-03-21 09:46:58,343 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 09:46:58,354 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-03-20'], host='localhost')
2025-03-21 09:46:58,369 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 09:47:01,832 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18327358 = 2025-03-22 21:18:00 UTC, 59.2% complete
2025-03-21 09:47:07,280 INFO [make_CHS_currents_file] running in process 2442342
2025-03-21 09:47:07,280 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:47:07,285 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5562
2025-03-21 09:47:07,285 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:47:13,615 INFO [make_plots] running in process 2442341
2025-03-21 09:47:13,617 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:47:13,623 INFO [make_plots] publishing log messages to tcp://*:5566
2025-03-21 09:47:13,647 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:47:13,675 INFO [make_plots] running in process 2442339
2025-03-21 09:47:13,698 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 09:47:13,722 INFO [make_plots] publishing log messages to tcp://*:5567
2025-03-21 09:47:13,739 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-03-21 09:47:15,160 INFO [make_plots] preparing render list for 2025-03-20 NEMO nowcast-blue comparison figures
2025-03-21 09:47:15,169 INFO [make_plots] preparing render list for 2025-03-21 NEMO nowcast-blue research figures
2025-03-21 09:47:15,342 INFO [make_plots] added SH_wind figure to 2025-03-20 NEMO nowcast-blue comparison render list
2025-03-21 09:47:15,346 INFO [make_plots] added Compare_VENUS_East figure to 2025-03-20 NEMO nowcast-blue comparison render list
2025-03-21 09:47:15,346 INFO [make_plots] added Compare_VENUS_Central figure to 2025-03-20 NEMO nowcast-blue comparison render list
2025-03-21 09:47:15,347 INFO [make_plots] starting to render nemo nowcast comparison 20mar25 figures
2025-03-21 09:47:18,452 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-blue research render list
2025-03-21 09:47:20,552 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-blue research render list
2025-03-21 09:47:20,553 INFO [make_plots] added Currents_sections_and_surface figure to 2025-03-21 NEMO nowcast-blue research render list
2025-03-21 09:47:20,558 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-03-21 NEMO nowcast-blue research render list
2025-03-21 09:47:20,561 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-03-21 NEMO nowcast-blue research render list
2025-03-21 09:47:20,565 INFO [make_plots] starting to render nemo nowcast research 21mar25 figures
2025-03-21 09:48:09,328 INFO [make_CHS_currents_file] Made CHS currents file for 2025-03-21 for nowcast
2025-03-21 09:48:09,335 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-03-21 09:48:40,811 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots
    checklist = _render_figures(
        config, model, run_type, plot_type, dmy, fig_functions, test_figure_id
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 975, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1042, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/comparison/compare_venus_ctd.py", line 77, in make_figure
    plot_data = _prep_plot_data(
        node_name, grid_T_hr, dev_grid_T_hr, timezone, mesh_mask, dev_mesh_mask
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/comparison/compare_venus_ctd.py", line 168, in _prep_plot_data
    onc_data = data_tools.get_onc_data(
        "scalardata",
    ...<6 lines>...
        dateTo=data_tools.onc_datetime(model_time[-1], "utc"),
    )
  File "/SalishSeaCast/tools/SalishSeaTools/salishsea_tools/data_tools.py", line 333, in get_onc_data
    response.raise_for_status()
    ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/requests/models.py", line 1024, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 400 Client Error:  for url: https://data.oceannetworks.ca/api/scalardata?method=getByLocation&token=8efe44eb-0fb5-4413-bba3-ae3dbe0fbb1d&locationCode=SEVIP&deviceCategoryCode=CTD&sensorCategoryCodes=salinity%2Ctemperature&dateFrom=2025-03-20T00:30:00.000Z&dateTo=2025-03-20T23:30:00.000Z
2025-03-21 09:49:35,614 INFO [make_plots] finished rendering nemo nowcast research 21mar25 figures
2025-03-21 09:49:35,616 INFO [make_plots] nemo research plots for 2025-03-21 nowcast completed
2025-03-21 09:49:35,636 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-03-21 09:52:01,833 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18327865 = 2025-03-23 02:56:00 UTC, 74.8% complete
2025-03-21 09:57:01,834 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18328357 = 2025-03-23 08:24:00 UTC, 90.0% complete
2025-03-21 10:02:01,836 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18328680 = 2025-03-23 11:59:20 UTC, 100.0% complete
2025-03-21 10:07:01,840 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-03-21 10:07:01,847 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-03-21 10:07:02,401 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-03-21'], host='arbutus.cloud-nowcast')
2025-03-21 10:07:02,414 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-03-21'], host='arbutus.cloud-nowcast')
2025-03-21 10:07:02,428 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:07:02,485 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:07:02,513 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:07:02,536 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:07:02,552 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:07:05,321 INFO [make_ww3_wind_file] running in process 12746
2025-03-21 10:07:05,321 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:05,323 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-03-21 10:07:05,328 INFO [make_ww3_current_file] running in process 12744
2025-03-21 10:07:05,329 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:05,331 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-03-21 10:07:05,334 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:05,336 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-03-21 forecast run
2025-03-21 10:07:05,351 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:05,353 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-03-21 forecast run
2025-03-21 10:07:08,600 INFO [upload_forcing] running in process 2442543
2025-03-21 10:07:08,602 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:08,603 INFO [upload_forcing] publishing log messages to tcp://*:5562
2025-03-21 10:07:08,605 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:08,632 INFO [download_results] running in process 2442545
2025-03-21 10:07:08,633 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:08,636 INFO [download_results] publishing log messages to tcp://*:5566
2025-03-21 10:07:08,637 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:08,645 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/21mar25 to /results/SalishSea/forecast.202111
2025-03-21 10:07:08,738 INFO [upload_forcing] running in process 2442542
2025-03-21 10:07:08,739 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:08,740 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-03-21 10:07:08,741 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:08,744 INFO [upload_forcing] running in process 2442544
2025-03-21 10:07:08,748 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:08,754 INFO [upload_forcing] publishing log messages to tcp://*:5568
2025-03-21 10:07:08,754 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:08,772 INFO [upload_forcing] running in process 2442541
2025-03-21 10:07:08,777 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:08,779 INFO [upload_forcing] publishing log messages to tcp://*:5569
2025-03-21 10:07:08,780 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:09,943 INFO [upload_forcing] turbidity 2025-03-21 forcing files upload to optimum-hindcast completed
2025-03-21 10:07:09,954 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-03-21 10:07:09,956 INFO [upload_forcing] turbidity 2025-03-21 forcing files upload to orcinus-nowcast-agrif completed
2025-03-21 10:07:10,255 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-03-21 10:07:10,606 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:07:11,382 INFO [upload_forcing] turbidity 2025-03-21 forcing files upload to arbutus.cloud-nowcast completed
2025-03-21 10:07:11,386 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-03-21 10:07:11,692 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:07:12,772 INFO [upload_forcing] turbidity 2025-03-21 forcing files upload to robot.graham completed
2025-03-21 10:07:12,788 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-03-21 10:07:14,860 INFO [make_forcing_links] running in process 2442576
2025-03-21 10:07:14,889 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:14,908 INFO [make_forcing_links] publishing log messages to tcp://*:5562
2025-03-21 10:07:14,926 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:15,191 INFO [make_forcing_links] running in process 2442578
2025-03-21 10:07:15,213 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:15,216 INFO [make_forcing_links] publishing log messages to tcp://*:5567
2025-03-21 10:07:15,218 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:15,991 INFO [make_forcing_links] nowcast-agrif 2025-03-21 forcing file links on orcinus-nowcast-agrif created
2025-03-21 10:07:15,999 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-03-21 10:07:16,305 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:07:16,617 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-03-21 forecast run
2025-03-21 10:07:16,621 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-03-21 10:07:18,550 INFO [run_NEMO_agrif] running in process 2442593
2025-03-21 10:07:18,552 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:18,553 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5562
2025-03-21 10:07:18,554 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:24,169 INFO [make_forcing_links] nowcast-green 2025-03-21 forcing file links on arbutus.cloud-nowcast created
2025-03-21 10:07:24,181 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-03-21 10:07:24,458 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-03-21'], host='arbutus.cloud-nowcast')
2025-03-21 10:07:28,890 INFO [run_NEMO_agrif] job id for 21mar25nowcast-agrif: 11418268.orca2.ibb
2025-03-21 10:07:28,905 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2025-03-21 queued on orcinus-nowcast-agrif
2025-03-21 10:07:28,934 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2025-03-21 10:07:29,225 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11418268.orca2.ibb'], host='localhost')
2025-03-21 10:07:29,681 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/21mar25nowcast-green_2025-03-21T100728.417939-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-03-21 10:07:29,685 INFO [run_NEMO] nowcast-green NEMO run for 2025-03-21 on arbutus.cloud-nowcast started
2025-03-21 10:07:29,688 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-03-21 10:07:30,006 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-03-21 10:07:31,847 INFO [watch_NEMO_agrif] running in process 2442635
2025-03-21 10:07:31,849 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:07:31,850 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5562
2025-03-21 10:07:31,852 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:07:32,906 INFO [watch_NEMO_agrif] watching 21mar25nowcast-agrif job 11418268 on orcinus-nowcast-agrif
2025-03-21 10:07:33,140 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif job 11418268 is queued on orcinus-nowcast-agrif
2025-03-21 10:07:34,396 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-03-21 10:08:33,347 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif job 11418268 is queued on orcinus-nowcast-agrif
2025-03-21 10:09:34,390 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2025-03-21 10:09:39,291 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-03-21 forecast run
2025-03-21 10:09:39,302 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-03-21 10:09:39,487 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-03-21'], host='arbutus.cloud-nowcast')
2025-03-21 10:09:49,295 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-03-21 10:09:49,561 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-03-21 10:10:00,341 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-03-21 10:12:34,875 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18323439 = 2025-03-21 01:45:20 UTC, 7.3% complete
2025-03-21 10:14:34,860 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8344333 = 2025-03-21 02:48:00 UTC, 11.7% complete
2025-03-21 10:15:00,422 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 90, 20.8% complete
2025-03-21 10:17:34,607 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18323614 = 2025-03-21 03:42:00 UTC, 15.4% complete
2025-03-21 10:19:35,301 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8344590 = 2025-03-21 05:39:20 UTC, 23.6% complete
2025-03-21 10:20:00,060 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 198, 45.8% complete
2025-03-21 10:22:34,412 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18323790 = 2025-03-21 05:39:20 UTC, 23.6% complete
2025-03-21 10:24:35,774 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8344850 = 2025-03-21 08:32:40 UTC, 35.6% complete
2025-03-21 10:24:59,808 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 309, 71.5% complete
2025-03-21 10:27:34,382 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18323965 = 2025-03-21 07:36:00 UTC, 31.7% complete
2025-03-21 10:29:36,261 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8345111 = 2025-03-21 11:26:40 UTC, 47.7% complete
2025-03-21 10:30:00,224 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 414, 95.8% complete
2025-03-21 10:32:34,386 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18324141 = 2025-03-21 09:33:20 UTC, 39.8% complete
2025-03-21 10:34:36,767 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8345370 = 2025-03-21 14:19:20 UTC, 59.7% complete
2025-03-21 10:35:00,423 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-03-21 10:35:00,430 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-03-21 10:35:01,088 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:35:01,096 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-03-21'], host='arbutus.cloud-nowcast')
2025-03-21 10:35:04,995 INFO [download_wwatch3_results] running in process 2442828
2025-03-21 10:35:04,996 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:35:04,997 INFO [download_wwatch3_results] publishing log messages to tcp://*:5567
2025-03-21 10:35:04,997 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:35:11,274 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-03-21 10:35:11,695 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-03-21 10:35:20,338 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-03-21 10:37:34,433 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18324316 = 2025-03-21 11:30:00 UTC, 47.9% complete
2025-03-21 10:38:20,233 INFO [download_results] forecast 2025-03-21 results files from arbutus.cloud-nowcast downloaded
2025-03-21 10:38:20,258 INFO [manager] checklist updated with [results files] items from download_results worker
2025-03-21 10:38:20,592 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:38:29,373 INFO [make_CHS_currents_file] running in process 2442859
2025-03-21 10:38:29,394 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:38:29,412 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5566
2025-03-21 10:38:29,436 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:38:54,660 INFO [download_wwatch3_results] nowcast 2025-03-21 results files from arbutus.cloud-nowcast downloaded
2025-03-21 10:38:54,664 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-03-21 10:39:37,207 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8345630 = 2025-03-21 17:12:40 UTC, 71.7% complete
2025-03-21 10:39:57,492 INFO [make_CHS_currents_file] Made CHS currents file for 2025-03-21 for forecast
2025-03-21 10:39:57,516 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-03-21 10:39:57,826 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:40:01,120 INFO [update_forecast_datasets] running in process 2442893
2025-03-21 10:40:01,122 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:40:01,123 INFO [update_forecast_datasets] publishing log messages to tcp://*:5566
2025-03-21 10:40:01,127 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:40:01,131 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/21mar25 run
2025-03-21 10:40:01,558 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/21mar25 run: /results/SalishSea/rolling-forecasts/nemo
2025-03-21 10:40:01,571 INFO [update_forecast_datasets] nemo 2025-03-21 forecast rolling forecast datasets updated
2025-03-21 10:40:01,582 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-03-21 10:40:01,898 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-03-21 10:40:01,916 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:40:01,942 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:40:05,399 INFO [ping_erddap] running in process 2442898
2025-03-21 10:40:05,401 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:40:05,404 INFO [ping_erddap] publishing log messages to tcp://*:5566
2025-03-21 10:40:05,405 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:40:05,491 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-03-21 10:40:05,492 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-03-21 10:40:09,721 INFO [make_surface_current_tiles] running in process 2442901
2025-03-21 10:40:09,721 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:40:09,733 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5566
2025-03-21 10:40:09,743 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:40:09,818 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-03-21 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/21mar25
2025-03-21 10:40:18,897 INFO [make_plots] running in process 2442900
2025-03-21 10:40:18,897 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 10:40:18,901 INFO [make_plots] publishing log messages to tcp://*:5567
2025-03-21 10:40:18,902 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-03-21 10:40:20,053 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 93, 14.4% complete
2025-03-21 10:40:20,545 INFO [make_plots] preparing render list for 2025-03-21 NEMO forecast publish figures
2025-03-21 10:40:20,800 INFO [make_plots] added Website_thumbnail figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,826 INFO [make_plots] added Threshold_website figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,828 INFO [make_plots] added PA_tidal_predictions figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,828 INFO [make_plots] added Boundary Bay figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,829 INFO [make_plots] added Campbell River figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,830 INFO [make_plots] added Cherry Point figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,830 INFO [make_plots] added Friday Harbor figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,831 INFO [make_plots] added Halfmoon Bay figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,832 INFO [make_plots] added Nanaimo figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,833 INFO [make_plots] added Neah Bay figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,834 INFO [make_plots] added New Westminster figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,835 INFO [make_plots] added Patricia Bay figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,836 INFO [make_plots] added Point Atkinson figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,837 INFO [make_plots] added Port Renfrew figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,838 INFO [make_plots] added Sand Heads figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,839 INFO [make_plots] added Sandy Cove figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,840 INFO [make_plots] added Squamish figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,842 INFO [make_plots] added Victoria figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,843 INFO [make_plots] added Woodwards Landing figure to 2025-03-21 NEMO forecast publish render list
2025-03-21 10:40:20,845 INFO [make_plots] starting to render nemo forecast publish 21mar25 figures
2025-03-21 10:42:34,306 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18324491 = 2025-03-21 13:26:40 UTC, 56.0% complete
2025-03-21 10:44:37,807 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8345891 = 2025-03-21 20:06:40 UTC, 83.8% complete
2025-03-21 10:45:20,063 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 201, 31.0% complete
2025-03-21 10:47:34,233 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18324667 = 2025-03-21 15:24:00 UTC, 64.2% complete
2025-03-21 10:49:38,340 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8346150 = 2025-03-21 22:59:20 UTC, 95.8% complete
2025-03-21 10:50:20,084 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 309, 47.7% complete
2025-03-21 10:52:34,248 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18324843 = 2025-03-21 17:21:20 UTC, 72.3% complete
2025-03-21 10:54:38,968 INFO [watch_NEMO_agrif] 21mar25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8346240 = 2025-03-21 23:59:20 UTC, 100.0% complete
2025-03-21 10:55:20,088 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 414, 63.9% complete
2025-03-21 10:57:01,520 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-03-21 10:57:01,616 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-03-21 10:57:34,827 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18325018 = 2025-03-21 19:18:00 UTC, 80.5% complete
2025-03-21 10:58:39,442 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 10:59:39,564 INFO [watch_NEMO_agrif] NEMO AGRIF run 11418268.orca2.ibb on orcinus-nowcast-agrif completed
2025-03-21 10:59:39,576 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2025-03-21 10:59:40,920 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 10:59:50,643 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/21mar25
2025-03-21 11:00:14,204 INFO [download_results] running in process 2445125
2025-03-21 11:00:14,205 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:00:14,222 INFO [download_results] publishing log messages to tcp://*:5562
2025-03-21 11:00:14,295 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:00:14,367 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/21mar25 to /results/SalishSea/nowcast-agrif.201702
2025-03-21 11:00:20,120 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 516, 79.6% complete
2025-03-21 11:01:04,104 INFO [download_results] nowcast-agrif 2025-03-21 results files from orcinus-nowcast-agrif downloaded
2025-03-21 11:01:04,129 INFO [manager] checklist updated with [results files] items from download_results worker
2025-03-21 11:01:04,497 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:01:12,888 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-03-21 11:01:12,892 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-03-21 11:01:41,669 INFO [make_plots] running in process 2445142
2025-03-21 11:01:41,669 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:01:41,709 INFO [make_plots] publishing log messages to tcp://*:5562
2025-03-21 11:01:41,727 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:01:42,936 INFO [make_plots] preparing render list for 2025-03-21 NEMO nowcast-agrif research figures
2025-03-21 11:01:42,937 INFO [make_plots] added baynes_sound_surface figure to 2025-03-21 NEMO nowcast-agrif research render list
2025-03-21 11:01:42,938 INFO [make_plots] starting to render nemo nowcast-agrif research 21mar25 figures
2025-03-21 11:01:48,820 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:02:05,964 INFO [make_plots] finished rendering nemo nowcast-agrif research 21mar25 figures
2025-03-21 11:02:06,000 INFO [make_plots] nemo research plots for 2025-03-21 nowcast-agrif completed
2025-03-21 11:02:06,038 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-03-21 11:02:34,260 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18325194 = 2025-03-21 21:15:20 UTC, 88.6% complete
2025-03-21 11:02:59,914 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:03:36,376 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:04:13,421 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:04:50,278 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:05:20,098 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 615, 94.9% complete
2025-03-21 11:05:21,282 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:05:58,464 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:06:34,050 WARNING [make_plots] invalid station number for Squamish station: None
2025-03-21 11:06:34,050 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-03-21 11:07:09,138 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:07:34,264 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18325370 = 2025-03-21 23:12:40 UTC, 96.8% complete
2025-03-21 11:07:45,790 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-03-16 00:00:00Z to 2025-03-23 11:55:00Z
2025-03-21 11:08:21,340 INFO [make_plots] finished rendering nemo forecast publish 21mar25 figures
2025-03-21 11:08:21,343 INFO [make_plots] nemo publish plots for 2025-03-21 forecast completed
2025-03-21 11:08:21,351 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-03-21 11:08:21,679 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:08:34,441 INFO [make_feeds] running in process 2445255
2025-03-21 11:08:34,441 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:08:34,498 INFO [make_feeds] publishing log messages to tcp://*:5562
2025-03-21 11:08:34,543 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:09:06,078 INFO [make_feeds] ATOM feeds for 2025-03-21 forecast run completed
2025-03-21 11:09:06,104 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-03-21 11:09:43,259 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/21mar25
2025-03-21 11:09:43,309 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-03-21 forecast into /results/nowcast-sys/figures/surface_currents/forecast/21mar25
2025-03-21 11:09:43,326 INFO [make_surface_current_tiles] surface current tile figures for 2025-03-21 forecast completed
2025-03-21 11:09:43,956 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-03-21 11:10:20,103 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-03-21 11:10:20,128 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-03-21 11:10:20,931 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:10:24,724 INFO [download_wwatch3_results] running in process 2445290
2025-03-21 11:10:24,724 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:10:24,726 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-03-21 11:10:24,729 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:12:34,402 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18325440 = 2025-03-21 23:59:20 UTC, 100.0% complete
2025-03-21 11:13:00,418 INFO [download_wwatch3_results] forecast 2025-03-21 results files from arbutus.cloud-nowcast downloaded
2025-03-21 11:13:00,422 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-03-21 11:13:01,013 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:13:06,636 INFO [update_forecast_datasets] running in process 2445316
2025-03-21 11:13:06,636 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:13:06,637 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-03-21 11:13:06,638 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:13:06,639 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/21mar25 run
2025-03-21 11:13:06,671 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/21mar25 run
2025-03-21 11:13:06,991 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/21mar25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-03-21 11:13:07,017 INFO [update_forecast_datasets] wwatch3 2025-03-21 forecast rolling forecast datasets updated
2025-03-21 11:13:07,054 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-03-21 11:13:07,669 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-03-21 11:13:10,669 INFO [ping_erddap] running in process 2445321
2025-03-21 11:13:10,672 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:13:10,673 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-03-21 11:13:10,673 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:13:10,674 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-03-21 11:13:10,680 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-03-21 11:13:11,357 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:13:26,120 INFO [make_plots] running in process 2445326
2025-03-21 11:13:26,120 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:13:26,133 INFO [make_plots] publishing log messages to tcp://*:5562
2025-03-21 11:13:26,137 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:13:26,138 INFO [make_plots] preparing render list for 2025-03-21 WaveWatch3 forecast publish figures
2025-03-21 11:13:26,139 INFO [make_plots] added Halibut Bank figure to 2025-03-21 WaveWatch3 forecast publish render list
2025-03-21 11:13:26,139 INFO [make_plots] added Sentry Shoal figure to 2025-03-21 WaveWatch3 forecast publish render list
2025-03-21 11:13:26,140 INFO [make_plots] starting to render wwatch3 forecast publish 21mar25 figures
2025-03-21 11:13:27,782 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-03-21 11:13:56,360 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-03-21 11:14:12,875 INFO [make_plots] finished rendering wwatch3 forecast publish 21mar25 figures
2025-03-21 11:14:12,876 INFO [make_plots] wwatch3 publish plots for 2025-03-21 forecast completed
2025-03-21 11:14:12,879 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-03-21 11:17:34,294 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-03-21 11:17:34,302 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-03-21 11:17:35,239 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:17:40,455 INFO [download_results] running in process 2445381
2025-03-21 11:17:40,456 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:17:40,457 INFO [download_results] publishing log messages to tcp://*:5562
2025-03-21 11:17:40,458 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:17:40,458 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/21mar25 to /results2/SalishSea/nowcast-green.202111
2025-03-21 11:51:13,846 INFO [download_results] nowcast-green 2025-03-21 results files from arbutus.cloud-nowcast downloaded
2025-03-21 11:51:13,889 INFO [manager] checklist updated with [results files] items from download_results worker
2025-03-21 11:51:14,523 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:51:14,541 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-03-21 11:51:14,585 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:51:14,643 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:51:14,654 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-03-21'], host='localhost')
2025-03-21 11:51:23,401 INFO [ping_erddap] running in process 2445588
2025-03-21 11:51:23,408 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:51:23,411 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-03-21 11:51:23,414 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:51:23,434 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-03-21 11:51:23,470 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-03-21 11:51:32,634 INFO [make_averaged_dataset] running in process 2445592
2025-03-21 11:51:32,635 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:51:32,700 INFO [make_averaged_dataset] publishing log messages to tcp://*:5562
2025-03-21 11:51:32,761 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:51:32,763 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Mar-2025 physics
2025-03-21 11:51:32,769 INFO [make_averaged_dataset] running in process 2445590
2025-03-21 11:51:32,770 INFO [make_averaged_dataset] running in process 2445591
2025-03-21 11:51:32,772 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:51:32,775 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:51:32,777 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2025-03-21 11:51:32,778 INFO [make_averaged_dataset] publishing log messages to tcp://*:5567
2025-03-21 11:51:32,780 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:51:32,781 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:51:32,781 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Mar-2025 chemistry
2025-03-21 11:51:32,783 INFO [make_averaged_dataset] creating day-averaged dataset for 21-Mar-2025 biology
2025-03-21 11:51:42,944 INFO [make_plots] running in process 2445587
2025-03-21 11:51:42,966 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 11:51:42,977 INFO [make_plots] publishing log messages to tcp://*:5568
2025-03-21 11:51:42,987 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-03-21 11:51:44,704 INFO [make_plots] preparing render list for 2025-03-21 NEMO nowcast-green research figures
2025-03-21 11:51:51,565 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:51:53,180 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:51:54,443 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:51:55,646 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:51:56,887 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:51:58,067 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:51:59,403 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:01,069 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:02,675 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:04,215 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:05,765 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:07,024 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:10,064 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:10,067 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:10,068 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:10,069 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-03-21 NEMO nowcast-green research render list
2025-03-21 11:52:10,069 INFO [make_plots] starting to render nemo nowcast-green research 21mar25 figures
2025-03-21 11:52:15,576 INFO [make_averaged_dataset] day-averaged dataset for 21-Mar-2025 physics created
2025-03-21 11:52:15,581 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-03-21 11:52:44,693 INFO [make_averaged_dataset] day-averaged dataset for 21-Mar-2025 chemistry created
2025-03-21 11:52:44,713 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-03-21 11:53:00,136 INFO [make_averaged_dataset] day-averaged dataset for 21-Mar-2025 biology created
2025-03-21 11:53:00,208 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-03-21 13:16:07,046 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 325, in make_plots
    checklist = _render_figures(
        config, model, run_type, plot_type, dmy, fig_functions, test_figure_id
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 975, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 1042, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/time_series_plots.py", line 79, in make_figure
    _plot_timeseries(axl, plot_data.left_var, left_variable, theme)
    ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/research/time_series_plots.py", line 123, in _plot_timeseries
    ax.plot(plot_data.time, plot_data, color=theme.COLOURS["time series"][variable])
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/axes/_axes.py", line 1777, in plot
    lines = [*self._get_lines(self, *args, data=data, **kwargs)]
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/axes/_base.py", line 297, in __call__
    yield from self._plot_args(
               ~~~~~~~~~~~~~~~^
        axes, this, kwargs, ambiguous_fmt_datakey=ambiguous_fmt_datakey,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        return_kwargs=return_kwargs
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/axes/_base.py", line 484, in _plot_args
    y = _check_1d(xy[1])
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/cbook.py", line 1351, in _check_1d
    x = _unpack_to_numpy(x)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/matplotlib/cbook.py", line 2349, in _unpack_to_numpy
    return x.to_numpy()
           ~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/dataarray.py", line 831, in to_numpy
    return self.variable.to_numpy()
           ~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/namedarray/core.py", line 860, in to_numpy
    return to_numpy(self._data)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/namedarray/pycompat.py", line 106, in to_numpy
    data = data.get_duck_array()  # type: ignore[no-untyped-call]
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 835, in get_duck_array
    self._ensure_cached()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 832, in _ensure_cached
    self.array = as_indexable(self.array.get_duck_array())
                              ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 789, in get_duck_array
    return self.array.get_duck_array()
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 659, in get_duck_array
    array = array.get_duck_array()
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/coding/variables.py", line 81, in get_duck_array
    return self.func(self.array.get_duck_array())
                     ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 652, in get_duck_array
    array = self.array[self.key]
            ~~~~~~~~~~^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 103, in __getitem__
    return indexing.explicit_indexing_adapter(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 1013, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 116, in _getitem
    array = getitem(original_array, key)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/common.py", line 256, in robust_getitem
    return array[key]
           ~~~~~^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 5079, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 6051, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Malformed or inaccessible DAP2 DATADDS or DAP4 DAP response
2025-03-21 15:04:02,549 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/18/ to /results/forcing/atmospheric/continental2.5/GRIB/20250321/18/
2025-03-21 15:04:02,550 INFO [collect_weather] 2025-03-21 2.5km weather forecast 18 collection complete
2025-03-21 15:04:02,560 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-03-21 15:04:03,093 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['00', '1km'], host='localhost')
2025-03-21 15:04:03,109 INFO [manager] launching NextWorker(module='nowcast.workers.download_weather', args=['12', '1km'], host='localhost')
2025-03-21 15:04:03,117 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['00', '2.5km'], host='localhost')
2025-03-21 15:04:03,124 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['00', '--fcst-date', '2025-03-22'], host='localhost')
2025-03-21 15:04:03,814 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250321/18/
2025-03-21 15:04:03,816 INFO [crop_gribs] 2025-03-21 18 GRIBs cropping complete
2025-03-21 15:04:03,819 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-03-21 15:04:05,602 INFO [collect_weather] running in process 2447949
2025-03-21 15:04:05,604 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 15:04:05,605 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-03-21 15:04:05,606 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-03-21 15:04:05,607 INFO [download_weather] running in process 2447948
2025-03-21 15:04:05,608 INFO [download_weather] running in process 2447947
2025-03-21 15:04:05,609 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 15:04:05,610 INFO [download_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 15:04:05,610 INFO [download_weather] publishing log messages to tcp://*:5563
2025-03-21 15:04:05,613 INFO [download_weather] publishing log messages to tcp://*:5564
2025-03-21 15:04:05,614 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-03-21 15:04:05,615 INFO [download_weather] connected to skookum.eos.ubc.ca port 5555
2025-03-21 15:04:05,615 INFO [download_weather] downloading 12 1 km forecast GRIB2 files for 20250321
2025-03-21 15:04:05,616 INFO [download_weather] downloading 00 1 km forecast GRIB2 files for 20250321
2025-03-21 15:04:05,701 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/00/
2025-03-21 15:04:07,218 INFO [crop_gribs] running in process 2447950
2025-03-21 15:04:07,219 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-03-21 15:04:07,220 INFO [crop_gribs] publishing log messages to tcp://*:5566
2025-03-21 15:04:07,221 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-03-21 15:04:07,223 INFO [crop_gribs] cropping 2025-03-22 ECCC HRDPS 2.5km continental 00Z GRIB files to SalishSeaCast subdomain
2025-03-21 15:04:07,235 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250322/00/
2025-03-21 15:07:58,638 INFO [download_weather] 2025-03-21 1km weather forecast 00 downloads complete
2025-03-21 15:07:58,641 INFO [manager] checklist updated with [weather forecast] items from download_weather worker
2025-03-21 15:08:02,185 INFO [download_weather] 2025-03-21 1km weather forecast 12 downloads complete
2025-03-21 15:08:02,188 INFO [manager] checklist updated with [weather forecast] items from download_weather worker