2025-06-30 03:41:07,919 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.log
2025-06-30 03:41:07,926 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/nowcast.debug.log
2025-06-30 03:41:07,930 INFO [rotate_logs] log file rotated: /SalishSeaCast/logs/nowcast/checklist.log
2025-06-30 03:41:07,931 INFO [rotate_logs] log files rotated
2025-06-30 03:41:07,935 INFO [manager] checklist updated with [log rotation] items from rotate_logs worker
2025-06-30 03:43:59,884 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast2/29jun25
2025-06-30 03:43:59,907 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-06-29 forecast2 into /results/nowcast-sys/figures/surface_currents/forecast2/29jun25
2025-06-30 03:43:59,908 INFO [make_surface_current_tiles] surface current tile figures for 2025-06-29 forecast2 completed
2025-06-30 03:44:00,208 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-06-30 03:45:16,607 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 399, 73.9% complete
2025-06-30 03:50:16,614 INFO [watch_ww3] forecast2 on arbutus.cloud-nowcast: timestep: 498, 92.2% complete
2025-06-30 03:55:16,618 INFO [watch_ww3] forecast2 WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-30 03:55:16,663 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-30 03:55:17,064 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast2', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 03:55:18,371 INFO [download_wwatch3_results] running in process 1230461
2025-06-30 03:55:18,373 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 03:55:18,374 INFO [download_wwatch3_results] publishing log messages to tcp://*:5562
2025-06-30 03:55:18,375 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-30 03:56:21,501 INFO [download_wwatch3_results] forecast2 2025-06-30 results files from arbutus.cloud-nowcast downloaded
2025-06-30 03:56:21,505 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-30 03:56:21,662 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast2', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 03:56:22,994 INFO [update_forecast_datasets] running in process 1230484
2025-06-30 03:56:22,996 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 03:56:22,997 INFO [update_forecast_datasets] publishing log messages to tcp://*:5562
2025-06-30 03:56:22,998 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-30 03:56:22,999 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast2/30jun25 run
2025-06-30 03:56:23,003 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast2/30jun25 run
2025-06-30 03:56:23,004 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast2/30jun25 run
2025-06-30 03:56:44,629 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast2/30jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-30 03:56:44,629 INFO [update_forecast_datasets] wwatch3 2025-06-30 forecast2 rolling forecast datasets updated
2025-06-30 03:56:44,633 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-30 03:56:44,789 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-30 03:56:45,831 INFO [ping_erddap] running in process 1230503
2025-06-30 03:56:45,833 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 03:56:45,834 INFO [ping_erddap] publishing log messages to tcp://*:5562
2025-06-30 03:56:45,835 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-30 03:56:45,837 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-30 03:56:45,839 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-30 03:56:45,991 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast2', 'publish', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 03:56:49,075 INFO [make_plots] running in process 1230508
2025-06-30 03:56:49,077 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 03:56:49,078 INFO [make_plots] publishing log messages to tcp://*:5562
2025-06-30 03:56:49,079 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-30 03:56:49,080 INFO [make_plots] preparing render list for 2025-06-30 WaveWatch3 forecast2 publish figures
2025-06-30 03:56:49,080 INFO [make_plots] added Halibut Bank figure to 2025-06-30 WaveWatch3 forecast2 publish render list
2025-06-30 03:56:49,081 INFO [make_plots] added Sentry Shoal figure to 2025-06-30 WaveWatch3 forecast2 publish render list
2025-06-30 03:56:49,082 INFO [make_plots] starting to render wwatch3 forecast2 publish 30jun25 figures
2025-06-30 03:56:49,564 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-30 04:17:22,308 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/tenacity/__init__.py", line 478, in __call__
    result = fn(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/wwatch3/wave_height_period.py", line 153, in _plot_wave_height_time_series
    plot_data.wwatch3.wave_height.plot(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        ax=ax,
        ^^^^^^
    ...<3 lines>...
        alpha=0.8,
        ^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/plot/accessor.py", line 48, in __call__
    return dataarray_plot.plot(self._da, **kwargs)
           ~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/plot/dataarray_plot.py", line 271, in plot
    ).compute()
      ~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/dataarray.py", line 1206, in compute
    return new.load(**kwargs)
           ~~~~~~~~^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/dataarray.py", line 1174, in load
    ds = self._to_temp_dataset().load(**kwargs)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/dataset.py", line 910, in load
    v.load()
    ~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/variable.py", line 1006, in load
    self._data = to_duck_array(self._data, **kwargs)
                 ~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/namedarray/pycompat.py", line 134, in to_duck_array
    return data.get_duck_array()  # type: ignore[no-untyped-call, no-any-return]
           ~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 835, in get_duck_array
    self._ensure_cached()
    ~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 832, in _ensure_cached
    self.array = as_indexable(self.array.get_duck_array())
                              ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 789, in get_duck_array
    return self.array.get_duck_array()
           ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 659, in get_duck_array
    array = array.get_duck_array()
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/coding/variables.py", line 81, in get_duck_array
    return self.func(self.array.get_duck_array())
                     ~~~~~~~~~~~~~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 652, in get_duck_array
    array = self.array[self.key]
            ~~~~~~~~~~^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 103, in __getitem__
    return indexing.explicit_indexing_adapter(
           ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^
        key, self.shape, indexing.IndexingSupport.OUTER, self._getitem
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/core/indexing.py", line 1013, in explicit_indexing_adapter
    result = raw_indexing_method(raw_key.tuple)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 116, in _getitem
    array = getitem(original_array, key)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/common.py", line 256, in robust_getitem
    return array[key]
           ~~~~~^^^^^
  File "src/netCDF4/_netCDF4.pyx", line 5079, in netCDF4._netCDF4.Variable.__getitem__
  File "src/netCDF4/_netCDF4.pyx", line 6051, in netCDF4._netCDF4.Variable._get
  File "src/netCDF4/_netCDF4.pyx", line 2164, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: NetCDF: Access failure

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 232, in make_plots
    checklist = _render_figures(
        config, model, run_type, plot_type, dmy, fig_functions, test_figure_id
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 766, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 833, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/wwatch3/wave_height_period.py", line 74, in make_figure
    _plot_wave_height_time_series(ax_sig_height, plot_data, theme)
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/tenacity/__init__.py", line 336, in wrapped_f
    return copy(f, *args, **kw)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/tenacity/__init__.py", line 475, in __call__
    do = self.iter(retry_state=retry_state)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/tenacity/__init__.py", line 376, in iter
    result = action(retry_state)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/tenacity/__init__.py", line 419, in exc_check
    raise retry_exc from fut.exception()
tenacity.RetryError: RetryError[<Future at 0x7f7e3f3fb7a0 state=finished raised RuntimeError>]
2025-06-30 08:51:03,458 INFO [collect_weather] finished collecting files from /SalishSeaCast/datamart/hrdps-continental/12/ to /results/forcing/atmospheric/continental2.5/GRIB/20250630/12/
2025-06-30 08:51:03,459 INFO [collect_weather] 2025-06-30 2.5km weather forecast 12 collection complete
2025-06-30 08:51:03,462 INFO [manager] checklist updated with [weather forecast] items from collect_weather worker
2025-06-30 08:51:03,811 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SkagitMountVernon', '--data-date', '2025-06-29'], host='localhost')
2025-06-30 08:51:03,818 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'SnohomishMonroe', '--data-date', '2025-06-29'], host='localhost')
2025-06-30 08:51:03,822 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'NisquallyMcKenna', '--data-date', '2025-06-29'], host='localhost')
2025-06-30 08:51:03,827 INFO [manager] launching NextWorker(module='nowcast.workers.collect_river_data', args=['USGS', 'GreenwaterGreenwater', '--data-date', '2025-06-29'], host='localhost')
2025-06-30 08:51:03,830 INFO [manager] launching NextWorker(module='nowcast.workers.make_turbidity_file', args=[], host='localhost')
2025-06-30 08:51:03,835 INFO [manager] launching NextWorker(module='nowcast.workers.collect_NeahBay_ssh', args=['06'], host='localhost')
2025-06-30 08:51:03,840 INFO [manager] launching NextWorker(module='nowcast.workers.download_live_ocean', args=[], host='localhost')
2025-06-30 08:51:03,845 INFO [manager] launching NextWorker(module='nowcast.workers.collect_weather', args=['18', '2.5km'], host='localhost')
2025-06-30 08:51:03,849 INFO [manager] launching NextWorker(module='nowcast.workers.crop_gribs', args=['18'], host='localhost')
2025-06-30 08:51:04,318 INFO [crop_gribs] finished cropping ECCC grib files to SalishSeaCast subdomain in /results/forcing/atmospheric/continental2.5/GRIB/20250630/12/
2025-06-30 08:51:04,319 INFO [crop_gribs] 2025-06-30 12 GRIBs cropping complete
2025-06-30 08:51:04,322 INFO [manager] checklist updated with [weather forecast] items from crop_gribs worker
2025-06-30 08:51:04,890 INFO [manager] launching NextWorker(module='nowcast.workers.grib_to_netcdf', args=['nowcast+'], host='localhost')
2025-06-30 08:51:05,476 INFO [collect_weather] running in process 1234574
2025-06-30 08:51:05,477 INFO [collect_weather] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:05,478 INFO [collect_weather] publishing log messages to tcp://*:5562
2025-06-30 08:51:05,479 INFO [collect_weather] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:05,488 INFO [collect_NeahBay_ssh] running in process 1234572
2025-06-30 08:51:05,489 INFO [collect_NeahBay_ssh] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:05,491 INFO [collect_NeahBay_ssh] publishing log messages to tcp://*:5563
2025-06-30 08:51:05,491 INFO [collect_NeahBay_ssh] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:05,492 INFO [collect_NeahBay_ssh] collecting Neah Bay ssh 06Z obs/forecast for 2025-06-30
2025-06-30 08:51:05,495 INFO [collect_weather] starting to watch for files in /SalishSeaCast/datamart/hrdps-continental/18/
2025-06-30 08:51:05,624 INFO [download_live_ocean] running in process 1234573
2025-06-30 08:51:05,625 INFO [download_live_ocean] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:05,626 INFO [download_live_ocean] publishing log messages to tcp://*:5564
2025-06-30 08:51:05,627 INFO [download_live_ocean] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:05,628 INFO [download_live_ocean] downloading Salish Sea western boundary day-averaged LiveOcean file for 2025-06-30
2025-06-30 08:51:05,958 INFO [collect_river_data] running in process 1234570
2025-06-30 08:51:05,959 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:05,960 INFO [collect_river_data] publishing log messages to tcp://*:5565
2025-06-30 08:51:05,960 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:05,961 INFO [collect_river_data] Collecting USGS GreenwaterGreenwater river data for 2025-06-29
2025-06-30 08:51:05,967 INFO [collect_river_data] running in process 1234568
2025-06-30 08:51:05,967 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:05,968 INFO [collect_river_data] publishing log messages to tcp://*:5566
2025-06-30 08:51:05,969 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:05,970 INFO [collect_river_data] Collecting USGS SnohomishMonroe river data for 2025-06-29
2025-06-30 08:51:06,042 INFO [collect_river_data] running in process 1234569
2025-06-30 08:51:06,043 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:06,044 INFO [collect_river_data] publishing log messages to tcp://*:5567
2025-06-30 08:51:06,045 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:06,045 INFO [collect_river_data] Collecting USGS NisquallyMcKenna river data for 2025-06-29
2025-06-30 08:51:06,067 INFO [make_turbidity_file] running in process 1234571
2025-06-30 08:51:06,068 INFO [make_turbidity_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:06,068 INFO [make_turbidity_file] publishing log messages to tcp://*:5568
2025-06-30 08:51:06,069 INFO [make_turbidity_file] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:06,070 INFO [make_turbidity_file] Creating Fraser River turbidity forcing file for 2025-06-30
2025-06-30 08:51:06,079 INFO [collect_river_data] running in process 1234566
2025-06-30 08:51:06,080 INFO [collect_river_data] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:06,081 INFO [collect_river_data] publishing log messages to tcp://*:5569
2025-06-30 08:51:06,081 INFO [collect_river_data] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:06,082 INFO [collect_river_data] Collecting USGS SkagitMountVernon river data for 2025-06-29
2025-06-30 08:51:06,356 INFO [download_live_ocean] downloaded 10183206 bytes from https://liveocean.apl.uw.edu/output/f2025.06.30/ubc.nc to /results/forcing/LiveOcean/downloaded/20250630/low_passed_UBC.nc
2025-06-30 08:51:06,357 INFO [download_live_ocean] Deflating in up to 1 concurrent sub-processes
2025-06-30 08:51:06,379 INFO [crop_gribs] running in process 1234575
2025-06-30 08:51:06,380 INFO [crop_gribs] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:06,381 INFO [crop_gribs] publishing log messages to tcp://*:5570
2025-06-30 08:51:06,382 INFO [crop_gribs] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:06,383 INFO [crop_gribs] cropping 2025-06-30 ECCC HRDPS 2.5km continental 18Z GRIB files to SalishSeaCast subdomain
2025-06-30 08:51:06,394 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12150800&startDT=2025-06-29&endDT=2025-06-29 "HTTP/1.1 200 "
2025-06-30 08:51:06,398 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12097500&startDT=2025-06-29&endDT=2025-06-29 "HTTP/1.1 200 "
2025-06-30 08:51:06,403 INFO [crop_gribs] starting to watch for ECCC grib files to crop in /results/forcing/atmospheric/continental2.5/GRIB/20250630/18/
2025-06-30 08:51:06,497 INFO [collect_river_data] Appended USGS GreenwaterGreenwater river average discharge for 2025-06-29 to: /results/forcing/rivers/observations/Greenwater_Greenwater_flow
2025-06-30 08:51:06,498 INFO [collect_river_data] USGS GreenwaterGreenwater river data collection for 2025-06-29 completed
2025-06-30 08:51:06,503 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-30 08:51:06,504 INFO [collect_river_data] Appended USGS SnohomishMonroe river average discharge for 2025-06-29 to: /results/forcing/rivers/observations/Snohomish_Monroe_flow
2025-06-30 08:51:06,504 INFO [collect_river_data] USGS SnohomishMonroe river data collection for 2025-06-29 completed
2025-06-30 08:51:06,551 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12089500&startDT=2025-06-29&endDT=2025-06-29 "HTTP/1.1 200 "
2025-06-30 08:51:06,594 INFO [collect_river_data] Appended USGS NisquallyMcKenna river average discharge for 2025-06-29 to: /results/forcing/rivers/observations/Nisqually_McKenna_flow
2025-06-30 08:51:06,595 INFO [collect_river_data] USGS NisquallyMcKenna river data collection for 2025-06-29 completed
2025-06-30 08:51:06,601 INFO [collect_river_data] HTTP Request: GET https://waterservices.usgs.gov/nwis/dv/?format=json&parameterCd=00060&sites=12200500&startDT=2025-06-29&endDT=2025-06-29 "HTTP/1.1 200 "
2025-06-30 08:51:06,606 INFO [collect_river_data] Appended USGS SkagitMountVernon river average discharge for 2025-06-29 to: /results/forcing/rivers/observations/Skagit_MountVernon_flow
2025-06-30 08:51:06,607 INFO [collect_river_data] USGS SkagitMountVernon river data collection for 2025-06-29 completed
2025-06-30 08:51:06,666 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-30 08:51:06,839 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-30 08:51:06,846 INFO [grib_to_netcdf] running in process 1234653
2025-06-30 08:51:06,848 INFO [grib_to_netcdf] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:06,849 INFO [grib_to_netcdf] publishing log messages to tcp://*:5571
2025-06-30 08:51:06,850 INFO [grib_to_netcdf] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:06,851 INFO [grib_to_netcdf] creating NEMO-atmos forcing files for 2025-06-30 nowcast and forecast runs
2025-06-30 08:51:07,002 INFO [manager] checklist updated with [river data] items from collect_river_data worker
2025-06-30 08:51:07,393 INFO [download_live_ocean] netCDF4 deflated /results/forcing/LiveOcean/downloaded/20250630/low_passed_UBC.nc
2025-06-30 08:51:07,394 INFO [download_live_ocean] 2025-06-30 Live Ocean file for Salish Sea western boundary downloaded
2025-06-30 08:51:07,396 INFO [manager] checklist updated with [Live Ocean products] items from download_live_ocean worker
2025-06-30 08:51:07,537 INFO [collect_NeahBay_ssh] 2025-06-30T00:00:00-07:00 Neah Bay ssh 06Z obs/forecast data collection complete
2025-06-30 08:51:07,722 INFO [manager] launching NextWorker(module='nowcast.workers.make_live_ocean_files', args=['--run-date', '2025-06-30'], host='localhost')
2025-06-30 08:51:07,731 INFO [manager] checklist updated with [Neah Bay ssh data] items from collect_NeahBay_ssh worker
2025-06-30 08:51:07,911 INFO [manager] launching NextWorker(module='nowcast.workers.make_ssh_files', args=['nowcast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 08:51:08,510 INFO [make_turbidity_file] 2025-06-30 Fraser River turbidity file creation complete
2025-06-30 08:51:08,512 INFO [manager] checklist updated with [Fraser River turbidity] items from make_turbidity_file worker
2025-06-30 08:51:09,910 INFO [make_live_ocean_files] running in process 1234706
2025-06-30 08:51:09,911 INFO [make_live_ocean_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:09,912 INFO [make_live_ocean_files] publishing log messages to tcp://*:5563
2025-06-30 08:51:09,913 INFO [make_live_ocean_files] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:09,913 INFO [make_live_ocean_files] Creating T&S western boundary conditions file from 2025-06-30 Live Ocean run
2025-06-30 08:51:10,586 INFO [make_ssh_files] running in process 1234708
2025-06-30 08:51:10,587 INFO [make_ssh_files] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:10,588 INFO [make_ssh_files] publishing log messages to tcp://*:5564
2025-06-30 08:51:10,590 INFO [make_ssh_files] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:10,591 INFO [make_ssh_files] building nowcast sea surface height boundary conditions file(s) from 2025-06-30 NOAA Neah Bay 06Z observation and forecast values
2025-06-30 08:51:12,258 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d26.nc
2025-06-30 08:51:12,432 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d27.nc
2025-06-30 08:51:12,618 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d28.nc
2025-06-30 08:51:12,808 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/obs/ssh_y2025m06d29.nc
2025-06-30 08:51:13,000 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m06d30.nc
2025-06-30 08:51:13,173 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d01.nc
2025-06-30 08:51:13,362 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d02.nc
2025-06-30 08:51:13,536 INFO [make_ssh_files] wrote sea surface height boundary file: /results/forcing/sshNeahBay/fcst/ssh_y2025m07d03.nc
2025-06-30 08:51:13,642 INFO [make_ssh_files] sea surface height boundary file for 2025-06-30 nowcast run created
2025-06-30 08:51:13,645 INFO [manager] checklist updated with [sea surface height forcing] items from make_ssh_files worker
2025-06-30 08:51:13,807 INFO [manager] launching NextWorker(module='nowcast.workers.make_runoff_file', args=['v202108'], host='localhost')
2025-06-30 08:51:13,812 INFO [manager] launching NextWorker(module='nowcast.workers.make_201702_runoff_file', args=[], host='localhost')
2025-06-30 08:51:15,094 INFO [make_201702_runoff_file] running in process 1234752
2025-06-30 08:51:15,095 INFO [make_201702_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:15,096 INFO [make_201702_runoff_file] publishing log messages to tcp://*:5564
2025-06-30 08:51:15,097 INFO [make_201702_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:15,121 INFO [make_201702_runoff_file] calculating runoff forcing for b201702 bathymetry
2025-06-30 08:51:15,215 INFO [make_201702_runoff_file] 2025-06-30 runoff file creation from Fraser at Hope and climatology elsewhere complete
2025-06-30 08:51:15,217 INFO [manager] checklist updated with [rivers forcing] items from make_201702_runoff_file worker
2025-06-30 08:51:15,690 INFO [make_runoff_file] running in process 1234750
2025-06-30 08:51:15,691 INFO [make_runoff_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:15,692 INFO [make_runoff_file] publishing log messages to tcp://*:5564
2025-06-30 08:51:15,693 INFO [make_runoff_file] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:15,694 INFO [make_runoff_file] calculating NEMO runoff forcing for v202108 bathymetry for 2025-06-29
2025-06-30 08:51:17,359 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/hrdps_y2025m06d30.nc
2025-06-30 08:51:18,469 INFO [make_runoff_file] stored NEMO runoff forcing for v202108 bathymetry for 2025-06-29: /results/forcing/rivers/R202108Dailies_y2025m06d29.nc
2025-06-30 08:51:18,470 INFO [make_runoff_file] 2025-06-29 runoff file creation completed
2025-06-30 08:51:18,472 INFO [manager] checklist updated with [rivers forcing] items from make_runoff_file worker
2025-06-30 08:51:24,687 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d01.nc
2025-06-30 08:51:28,850 INFO [grib_to_netcdf] created /results/forcing/atmospheric/continental2.5/nemo_forcing/fcst/hrdps_y2025m07d02.nc
2025-06-30 08:51:28,865 INFO [grib_to_netcdf] 2025-06-30 NEMO-atmos forcing file for nowcast+ created
2025-06-30 08:51:28,868 INFO [manager] checklist updated with [weather forcing] items from grib_to_netcdf worker
2025-06-30 08:51:50,887 INFO [make_live_ocean_files] Stored T&S western boundary conditions file: /results/forcing/LiveOcean/boundary_conditions/LiveOcean_v201905_y2025m06d30.nc
2025-06-30 08:51:50,888 INFO [make_live_ocean_files] 2025-06-30 Live Ocean western boundary conditions files created
2025-06-30 08:51:50,891 INFO [manager] checklist updated with [Live Ocean boundary conditions] items from make_live_ocean_files worker
2025-06-30 08:51:51,052 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['weather'], host='localhost')
2025-06-30 08:51:51,057 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'nowcast+'], host='localhost')
2025-06-30 08:51:51,061 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'nowcast+'], host='localhost')
2025-06-30 08:51:51,066 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'nowcast+'], host='localhost')
2025-06-30 08:51:51,071 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'nowcast+'], host='localhost')
2025-06-30 08:51:52,138 INFO [ping_erddap] running in process 1234802
2025-06-30 08:51:52,139 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:52,140 INFO [ping_erddap] publishing log messages to tcp://*:5563
2025-06-30 08:51:52,141 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:52,143 INFO [ping_erddap] weather ERDDAP dataset flag file(s) created
2025-06-30 08:51:52,144 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-30 08:51:52,232 INFO [upload_forcing] running in process 1234807
2025-06-30 08:51:52,233 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:52,234 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-30 08:51:52,235 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:52,239 INFO [upload_forcing] running in process 1234805
2025-06-30 08:51:52,240 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:52,242 INFO [upload_forcing] publishing log messages to tcp://*:5565
2025-06-30 08:51:52,243 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:52,250 INFO [upload_forcing] running in process 1234804
2025-06-30 08:51:52,251 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:52,253 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-30 08:51:52,254 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:52,255 INFO [upload_forcing] running in process 1234806
2025-06-30 08:51:52,256 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:51:52,257 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-30 08:51:52,258 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:51:54,265 INFO [upload_forcing] nowcast+ 2025-06-30 forcing files upload to optimum-hindcast completed
2025-06-30 08:51:54,269 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-30 08:51:54,447 INFO [upload_forcing] nowcast+ 2025-06-30 forcing files upload to orcinus-nowcast-agrif completed
2025-06-30 08:51:54,449 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-30 08:52:02,977 INFO [upload_forcing] nowcast+ 2025-06-30 forcing files upload to robot.graham completed
2025-06-30 08:52:02,982 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-30 08:52:04,069 INFO [upload_forcing] nowcast+ 2025-06-30 forcing files upload to arbutus.cloud-nowcast completed
2025-06-30 08:52:04,074 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-30 08:52:04,237 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast+', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 08:52:05,678 INFO [make_forcing_links] running in process 1234842
2025-06-30 08:52:05,681 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:52:05,682 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-06-30 08:52:05,683 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:52:06,841 INFO [make_forcing_links] nowcast+ 2025-06-30 forcing file links on arbutus.cloud-nowcast created
2025-06-30 08:52:06,845 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-30 08:52:07,011 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-30'], host='arbutus.cloud-nowcast')
2025-06-30 08:52:09,904 INFO [run_NEMO] running in process 18884
2025-06-30 08:52:09,905 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:52:09,907 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-30 08:52:09,965 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:52:10,771 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-30 08:52:11,080 INFO [run_NEMO] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/runs/30jun25nowcast_2025-06-30T085209.979596-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-30 08:52:11,093 INFO [run_NEMO] nowcast NEMO run for 2025-06-30 on arbutus.cloud-nowcast started
2025-06-30 08:52:11,097 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-30 08:52:11,257 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-06-30 08:52:13,193 INFO [watch_NEMO] running in process 19157
2025-06-30 08:52:13,194 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 08:52:13,195 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-30 08:52:13,196 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-30 08:52:13,311 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-30 08:57:13,305 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18541851 = 2025-06-30 04:33:20 UTC, 19.0% complete
2025-06-30 09:02:13,301 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18542338 = 2025-06-30 09:58:00 UTC, 41.5% complete
2025-06-30 09:07:13,301 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18542804 = 2025-06-30 15:08:40 UTC, 63.1% complete
2025-06-30 09:12:13,314 INFO [watch_NEMO] nowcast on arbutus.cloud-nowcast: timestep: 18543305 = 2025-06-30 20:42:40 UTC, 86.3% complete
2025-06-30 09:17:13,310 INFO [watch_NEMO] nowcast NEMO run on arbutus.cloud-nowcast completed
2025-06-30 09:17:13,314 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-30 09:17:13,670 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'ssh', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:17:13,677 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:17:14,921 INFO [download_results] running in process 1235217
2025-06-30 09:17:14,922 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:17:14,924 INFO [download_results] publishing log messages to tcp://*:5563
2025-06-30 09:17:14,925 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:17:14,926 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast/30jun25 to /results/SalishSea/nowcast-blue.202111
2025-06-30 09:17:14,942 INFO [make_forcing_links] running in process 1235216
2025-06-30 09:17:14,943 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:17:14,944 INFO [make_forcing_links] publishing log messages to tcp://*:5564
2025-06-30 09:17:14,945 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:17:15,893 INFO [make_forcing_links] ssh 2025-06-30 forcing file links on arbutus.cloud-nowcast created
2025-06-30 09:17:15,898 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-30 09:17:16,062 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-30'], host='arbutus.cloud-nowcast')
2025-06-30 09:17:18,150 INFO [run_NEMO] running in process 19573
2025-06-30 09:17:18,154 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:17:18,156 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-30 09:17:18,157 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:17:18,924 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-30 09:17:19,232 INFO [run_NEMO] forecast: launching /nemoShare/MEOPAR/nowcast-sys/runs/30jun25forecast_2025-06-30T091718.235257-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-30 09:17:19,247 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-30 09:17:19,252 INFO [run_NEMO] forecast NEMO run for 2025-06-30 on arbutus.cloud-nowcast started
2025-06-30 09:17:19,410 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-06-30 09:17:21,470 INFO [watch_NEMO] running in process 19846
2025-06-30 09:17:21,472 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:17:21,473 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-30 09:17:21,475 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:17:21,545 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-30 09:18:27,726 INFO [download_results] nowcast 2025-06-30 results files from arbutus.cloud-nowcast downloaded
2025-06-30 09:18:27,730 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-30 09:18:27,888 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'research', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:18:27,893 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast', 'comparison', '--run-date', '2025-06-29'], host='localhost')
2025-06-30 09:18:27,898 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['nowcast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:18:29,865 INFO [make_CHS_currents_file] running in process 1235247
2025-06-30 09:18:29,867 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:18:29,867 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5563
2025-06-30 09:18:29,869 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:18:30,767 INFO [make_plots] running in process 1235246
2025-06-30 09:18:30,768 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:18:30,770 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-30 09:18:30,771 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:18:30,777 INFO [make_plots] running in process 1235244
2025-06-30 09:18:30,795 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:18:30,812 INFO [make_plots] publishing log messages to tcp://*:5565
2025-06-30 09:18:30,818 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:18:31,141 INFO [make_plots] preparing render list for 2025-06-29 NEMO nowcast-blue comparison figures
2025-06-30 09:18:31,142 INFO [make_plots] preparing render list for 2025-06-30 NEMO nowcast-blue research figures
2025-06-30 09:18:31,196 INFO [make_plots] added SH_wind figure to 2025-06-29 NEMO nowcast-blue comparison render list
2025-06-30 09:18:31,197 INFO [make_plots] added Compare_VENUS_East figure to 2025-06-29 NEMO nowcast-blue comparison render list
2025-06-30 09:18:31,198 INFO [make_plots] added Compare_VENUS_Central figure to 2025-06-29 NEMO nowcast-blue comparison render list
2025-06-30 09:18:31,199 INFO [make_plots] starting to render nemo nowcast comparison 29jun25 figures
2025-06-30 09:18:32,358 INFO [make_plots] added salinity_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-blue research render list
2025-06-30 09:18:32,557 CRITICAL [make_plots] unhandled exception:
Traceback (most recent call last):
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/file_manager.py", line 211, in _acquire_with_cache_info
    file = self._cache[self._key]
           ~~~~~~~~~~~^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/lru_cache.py", line 56, in __getitem__
    value = self._cache[key]
            ~~~~~~~~~~~^^^^^
KeyError: [<class 'netCDF4._netCDF4.Dataset'>, ('https://salishsea.eos.ubc.ca/erddap/griddap/ubcSSaSurfaceAtmosphereFieldsV23-02',), 'r', (('clobber', True), ('diskless', False), ('format', 'NETCDF4'), ('persist', False)), '08930bdb-94ea-4b65-be7c-9bcf78713a23']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/SalishSeaCast/NEMO_Nowcast/nemo_nowcast/worker.py", line 387, in _do_work
    checklist = self.worker_func(
        self._parsed_args, self.config, self.tell_manager
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 232, in make_plots
    checklist = _render_figures(
        config, model, run_type, plot_type, dmy, fig_functions, test_figure_id
    )
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 766, in _render_figures
    fig = _calc_figure(fig_func, args, kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/workers/make_plots.py", line 833, in _calc_figure
    fig = fig_func(*args, **kwargs)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/comparison/sandheads_winds.py", line 76, in make_figure
    plot_data = _prep_plot_data(hrdps_dataset_url, run_type, run_date)
  File "/SalishSeaCast/SalishSeaNowcast/nowcast/figures/comparison/sandheads_winds.py", line 90, in _prep_plot_data
    hrdps = xarray.open_dataset(hrdps_dataset_url)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/api.py", line 679, in open_dataset
    backend_ds = backend.open_dataset(
        filename_or_obj,
    ...<2 lines>...
        **kwargs,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 666, in open_dataset
    store = NetCDF4DataStore.open(
        filename_or_obj,
    ...<8 lines>...
        autoclose=autoclose,
    )
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 452, in open
    return cls(manager, group=group, mode=mode, lock=lock, autoclose=autoclose)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 393, in __init__
    self.format = self.ds.data_model
                  ^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 461, in ds
    return self._acquire()
           ~~~~~~~~~~~~~^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/netCDF4_.py", line 455, in _acquire
    with self._manager.acquire_context(needs_lock) as root:
         ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/contextlib.py", line 141, in __enter__
    return next(self.gen)
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/file_manager.py", line 199, in acquire_context
    file, cached = self._acquire_with_cache_info(needs_lock)
                   ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
  File "/SalishSeaCast/nowcast-env/lib/python3.13/site-packages/xarray/backends/file_manager.py", line 217, in _acquire_with_cache_info
    file = self._opener(*self._args, **kwargs)
  File "src/netCDF4/_netCDF4.pyx", line 2521, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 2158, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -90] NetCDF: file not found: 'https://salishsea.eos.ubc.ca/erddap/griddap/ubcSSaSurfaceAtmosphereFieldsV23-02'
2025-06-30 09:18:33,114 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-blue research render list
2025-06-30 09:18:33,115 INFO [make_plots] added Currents_sections_and_surface figure to 2025-06-30 NEMO nowcast-blue research render list
2025-06-30 09:18:33,116 INFO [make_plots] added Currents_at_VENUS_Central figure to 2025-06-30 NEMO nowcast-blue research render list
2025-06-30 09:18:33,116 INFO [make_plots] added Currents_at_VENUS_East figure to 2025-06-30 NEMO nowcast-blue research render list
2025-06-30 09:18:33,117 INFO [make_plots] starting to render nemo nowcast research 30jun25 figures
2025-06-30 09:19:00,064 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-30 for nowcast
2025-06-30 09:19:00,067 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-30 09:19:36,491 INFO [make_plots] finished rendering nemo nowcast research 30jun25 figures
2025-06-30 09:19:36,493 INFO [make_plots] nemo research plots for 2025-06-30 nowcast completed
2025-06-30 09:19:36,508 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-30 09:22:21,583 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18544060 = 2025-07-01 05:06:00 UTC, 14.2% complete
2025-06-30 09:27:21,527 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18544526 = 2025-07-01 10:16:40 UTC, 28.6% complete
2025-06-30 09:32:21,532 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18545031 = 2025-07-01 15:53:20 UTC, 44.1% complete
2025-06-30 09:37:21,539 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18545520 = 2025-07-01 21:19:20 UTC, 59.2% complete
2025-06-30 09:42:21,545 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18546030 = 2025-07-02 02:59:20 UTC, 75.0% complete
2025-06-30 09:47:21,585 INFO [watch_NEMO] forecast on arbutus.cloud-nowcast: timestep: 18546526 = 2025-07-02 08:30:00 UTC, 90.3% complete
2025-06-30 09:52:21,554 INFO [watch_NEMO] forecast NEMO run on arbutus.cloud-nowcast completed
2025-06-30 09:52:21,557 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-30 09:52:21,928 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_wind_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-30'], host='arbutus.cloud-nowcast')
2025-06-30 09:52:21,933 INFO [manager] launching NextWorker(module='nowcast.workers.make_ww3_current_file', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-30'], host='arbutus.cloud-nowcast')
2025-06-30 09:52:21,938 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['arbutus.cloud-nowcast', 'turbidity', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:52:21,942 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['orcinus-nowcast-agrif', 'turbidity', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:52:21,948 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['robot.graham', 'turbidity', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:52:21,952 INFO [manager] launching NextWorker(module='nowcast.workers.upload_forcing', args=['optimum-hindcast', 'turbidity', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:52:21,958 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:52:23,188 INFO [upload_forcing] running in process 1238898
2025-06-30 09:52:23,189 INFO [upload_forcing] running in process 1238900
2025-06-30 09:52:23,191 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:23,192 INFO [upload_forcing] publishing log messages to tcp://*:5564
2025-06-30 09:52:23,193 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:23,195 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:23,196 INFO [upload_forcing] publishing log messages to tcp://*:5563
2025-06-30 09:52:23,197 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:23,205 INFO [download_results] running in process 1238902
2025-06-30 09:52:23,206 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:23,207 INFO [download_results] publishing log messages to tcp://*:5565
2025-06-30 09:52:23,208 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:23,209 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/forecast/30jun25 to /results/SalishSea/forecast.202111
2025-06-30 09:52:23,237 INFO [upload_forcing] running in process 1238899
2025-06-30 09:52:23,238 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:23,239 INFO [upload_forcing] publishing log messages to tcp://*:5566
2025-06-30 09:52:23,240 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:23,268 INFO [upload_forcing] running in process 1238901
2025-06-30 09:52:23,269 INFO [upload_forcing] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:23,271 INFO [upload_forcing] publishing log messages to tcp://*:5567
2025-06-30 09:52:23,272 INFO [upload_forcing] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:23,926 INFO [upload_forcing] turbidity 2025-06-30 forcing files upload to orcinus-nowcast-agrif completed
2025-06-30 09:52:23,930 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-30 09:52:23,945 INFO [upload_forcing] turbidity 2025-06-30 forcing files upload to optimum-hindcast completed
2025-06-30 09:52:24,090 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:52:24,098 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-30 09:52:24,390 INFO [upload_forcing] turbidity 2025-06-30 forcing files upload to arbutus.cloud-nowcast completed
2025-06-30 09:52:24,394 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-30 09:52:24,570 INFO [manager] launching NextWorker(module='nowcast.workers.make_forcing_links', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:52:25,498 INFO [make_ww3_wind_file] running in process 20516
2025-06-30 09:52:25,499 INFO [make_ww3_wind_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:25,501 INFO [make_ww3_wind_file] publishing log messages to tcp://*:5570
2025-06-30 09:52:25,512 INFO [make_forcing_links] running in process 1238937
2025-06-30 09:52:25,514 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:25,515 INFO [make_forcing_links] publishing log messages to tcp://*:5563
2025-06-30 09:52:25,516 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:25,655 INFO [make_ww3_current_file] running in process 20517
2025-06-30 09:52:25,657 INFO [make_ww3_current_file] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:25,659 INFO [make_ww3_current_file] publishing log messages to tcp://*:5571
2025-06-30 09:52:25,987 INFO [make_forcing_links] running in process 1238940
2025-06-30 09:52:25,988 INFO [make_forcing_links] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:25,989 INFO [make_forcing_links] publishing log messages to tcp://*:5566
2025-06-30 09:52:25,990 INFO [make_forcing_links] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:26,292 INFO [make_forcing_links] nowcast-agrif 2025-06-30 forcing file links on orcinus-nowcast-agrif created
2025-06-30 09:52:26,296 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-30 09:52:26,470 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO_agrif', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:52:27,265 INFO [make_forcing_links] nowcast-green 2025-06-30 forcing file links on arbutus.cloud-nowcast created
2025-06-30 09:52:27,269 INFO [manager] checklist updated with [forcing links] items from make_forcing_links worker
2025-06-30 09:52:27,439 INFO [manager] launching NextWorker(module='nowcast.workers.run_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-30'], host='arbutus.cloud-nowcast')
2025-06-30 09:52:27,697 INFO [upload_forcing] turbidity 2025-06-30 forcing files upload to robot.graham completed
2025-06-30 09:52:27,701 INFO [manager] checklist updated with [forcing upload] items from upload_forcing worker
2025-06-30 09:52:27,833 INFO [run_NEMO_agrif] running in process 1238972
2025-06-30 09:52:27,834 INFO [run_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:27,836 INFO [run_NEMO_agrif] publishing log messages to tcp://*:5563
2025-06-30 09:52:27,838 INFO [run_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:28,104 INFO [make_ww3_wind_file] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:28,106 INFO [make_ww3_wind_file] Creating wwatch3 wind forcing file for 2025-06-30 forecast run
2025-06-30 09:52:28,118 INFO [make_ww3_current_file] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:28,118 INFO [make_ww3_current_file] Creating wwatch3 currents forcing file for 2025-06-30 forecast run
2025-06-30 09:52:30,005 INFO [run_NEMO] running in process 20832
2025-06-30 09:52:30,006 INFO [run_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:30,008 INFO [run_NEMO] publishing log messages to tcp://*:5556
2025-06-30 09:52:30,009 INFO [run_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:32,003 WARNING [run_NEMO] There are uncommitted changes in /nemoShare/MEOPAR/nowcast-sys/SS-run-sets
2025-06-30 09:52:32,795 INFO [run_NEMO] nowcast-green: launching /nemoShare/MEOPAR/nowcast-sys/runs/30jun25nowcast-green_2025-06-30T095230.091541-0700/SalishSeaNEMO.sh on arbutus.cloud-nowcast
2025-06-30 09:52:32,823 INFO [manager] checklist updated with [NEMO run] items from run_NEMO worker
2025-06-30 09:52:32,835 INFO [run_NEMO] nowcast-green NEMO run for 2025-06-30 on arbutus.cloud-nowcast started
2025-06-30 09:52:32,990 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO', args=['arbutus.cloud-nowcast', 'nowcast-green'], host='arbutus.cloud-nowcast')
2025-06-30 09:52:36,415 INFO [watch_NEMO] running in process 21428
2025-06-30 09:52:36,419 INFO [watch_NEMO] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:36,420 INFO [watch_NEMO] publishing log messages to tcp://*:5557
2025-06-30 09:52:36,422 INFO [watch_NEMO] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:36,815 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: time.step not found; continuing to watch...
2025-06-30 09:52:37,536 INFO [manager] checklist updated with [WW3 wind forcing] items from make_ww3_wind_file worker
2025-06-30 09:52:37,536 INFO [make_ww3_wind_file] wwatch3 wind forcing file created on arbutus.cloud-nowcast for 2025-06-30 forecast run
2025-06-30 09:52:44,948 INFO [run_NEMO_agrif] job id for 30jun25nowcast-agrif: 11433312.orca2.ibb
2025-06-30 09:52:44,949 INFO [run_NEMO_agrif] NEMO nowcast-agrif run for 2025-06-30 queued on orcinus-nowcast-agrif
2025-06-30 09:52:44,956 INFO [manager] checklist updated with [NEMO run] items from run_NEMO_agrif worker
2025-06-30 09:52:45,145 INFO [manager] launching NextWorker(module='nowcast.workers.watch_NEMO_agrif', args=['orcinus-nowcast-agrif', '11433312.orca2.ibb'], host='localhost')
2025-06-30 09:52:46,545 INFO [watch_NEMO_agrif] running in process 1239088
2025-06-30 09:52:46,546 INFO [watch_NEMO_agrif] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:52:46,547 INFO [watch_NEMO_agrif] publishing log messages to tcp://*:5563
2025-06-30 09:52:46,549 INFO [watch_NEMO_agrif] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:52:47,444 INFO [watch_NEMO_agrif] watching 30jun25nowcast-agrif job 11433312 on orcinus-nowcast-agrif
2025-06-30 09:52:47,680 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif job 11433312 is queued on orcinus-nowcast-agrif
2025-06-30 09:53:47,890 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif job 11433312 is queued on orcinus-nowcast-agrif
2025-06-30 09:54:03,661 INFO [download_results] forecast 2025-06-30 results files from arbutus.cloud-nowcast downloaded
2025-06-30 09:54:03,666 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-30 09:54:03,836 INFO [manager] launching NextWorker(module='nowcast.workers.make_CHS_currents_file', args=['forecast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:54:06,204 INFO [make_CHS_currents_file] running in process 1239217
2025-06-30 09:54:06,205 INFO [make_CHS_currents_file] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:54:06,206 INFO [make_CHS_currents_file] publishing log messages to tcp://*:5564
2025-06-30 09:54:06,207 INFO [make_CHS_currents_file] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:54:48,939 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: time.step not found; continuing to watch...
2025-06-30 09:54:51,689 INFO [make_CHS_currents_file] Made CHS currents file for 2025-06-30 for forecast
2025-06-30 09:54:51,693 INFO [manager] checklist updated with [CHS currents file] items from make_CHS_currents_file worker
2025-06-30 09:54:51,824 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['nemo', 'forecast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:54:53,012 INFO [update_forecast_datasets] running in process 1239244
2025-06-30 09:54:53,013 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:54:53,014 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-06-30 09:54:53,015 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:54:53,017 INFO [update_forecast_datasets] updating nemo forecast directory for forecast/30jun25 run
2025-06-30 09:54:53,306 INFO [update_forecast_datasets] updated nemo forecast directory for forecast/30jun25 run: /results/SalishSea/rolling-forecasts/nemo
2025-06-30 09:54:53,307 INFO [update_forecast_datasets] nemo 2025-06-30 forecast rolling forecast datasets updated
2025-06-30 09:54:53,309 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-30 09:54:53,480 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nemo-forecast'], host='localhost')
2025-06-30 09:54:53,485 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'forecast', 'publish', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:54:53,491 INFO [manager] launching NextWorker(module='nowcast.workers.make_surface_current_tiles', args=['forecast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 09:54:54,272 INFO [make_ww3_current_file] wwatch3 currents forcing file created on arbutus.cloud-nowcast for 2025-06-30 forecast run
2025-06-30 09:54:54,276 INFO [manager] checklist updated with [WW3 currents forcing] items from make_ww3_current_file worker
2025-06-30 09:54:54,426 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-30'], host='arbutus.cloud-nowcast')
2025-06-30 09:54:54,560 INFO [ping_erddap] running in process 1239249
2025-06-30 09:54:54,561 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:54:54,562 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-06-30 09:54:54,563 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:54:54,571 INFO [ping_erddap] nemo-forecast ERDDAP dataset flag file(s) created
2025-06-30 09:54:54,573 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-30 09:54:55,832 INFO [make_surface_current_tiles] running in process 1239252
2025-06-30 09:54:55,833 INFO [make_surface_current_tiles] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:54:55,834 INFO [make_surface_current_tiles] publishing log messages to tcp://*:5564
2025-06-30 09:54:55,835 INFO [make_surface_current_tiles] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:54:55,838 INFO [make_surface_current_tiles] starting rendering of tiles for 2025-06-30 forecast in 6 processes into /results/nowcast-sys/figures/surface_currents/forecast/30jun25
2025-06-30 09:54:56,510 INFO [run_ww3] running in process 21534
2025-06-30 09:54:56,515 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:54:56,516 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-06-30 09:54:56,518 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:54:56,520 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-06-30T095456.502362-0700
2025-06-30 09:54:56,523 INFO [run_ww3] nowcast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/nowcast_2025-06-30T095456.502362-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-06-30 09:54:56,529 INFO [run_ww3] nowcast WaveWatch3 run for 2025-06-30 on arbutus.cloud-nowcast started
2025-06-30 09:54:56,533 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-30 09:54:56,859 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'nowcast'], host='arbutus.cloud-nowcast')
2025-06-30 09:54:57,147 INFO [make_plots] running in process 1239251
2025-06-30 09:54:57,148 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:54:57,152 INFO [make_plots] publishing log messages to tcp://*:5565
2025-06-30 09:54:57,157 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:54:57,566 INFO [make_plots] preparing render list for 2025-06-30 NEMO forecast publish figures
2025-06-30 09:54:57,663 INFO [make_plots] added Website_thumbnail figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,665 INFO [make_plots] added Threshold_website figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,666 INFO [make_plots] added PA_tidal_predictions figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,667 INFO [make_plots] added Boundary Bay figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,667 INFO [make_plots] added Campbell River figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,668 INFO [make_plots] added Cherry Point figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,669 INFO [make_plots] added Friday Harbor figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,669 INFO [make_plots] added Halfmoon Bay figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,671 INFO [make_plots] added Nanaimo figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,673 INFO [make_plots] added Neah Bay figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,673 INFO [make_plots] added New Westminster figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,674 INFO [make_plots] added Patricia Bay figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,675 INFO [make_plots] added Point Atkinson figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,676 INFO [make_plots] added Port Renfrew figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,677 INFO [make_plots] added Sand Heads figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,678 INFO [make_plots] added Sandy Cove figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,679 INFO [make_plots] added Squamish figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,681 INFO [make_plots] added Victoria figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,682 INFO [make_plots] added Woodwards Landing figure to 2025-06-30 NEMO forecast publish render list
2025-06-30 09:54:57,683 INFO [make_plots] starting to render nemo forecast publish 30jun25 figures
2025-06-30 09:54:59,123 INFO [watch_ww3] running in process 21621
2025-06-30 09:54:59,124 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 09:54:59,125 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-30 09:54:59,126 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-30 09:54:59,162 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-30 09:57:36,813 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18541582 = 2025-06-30 01:34:00 UTC, 6.5% complete
2025-06-30 09:59:49,384 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8562407 = 2025-06-30 01:50:40 UTC, 7.7% complete
2025-06-30 09:59:59,162 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 72, 16.7% complete
2025-06-30 10:02:28,712 WARNING [make_plots] invalid station number for Boundary Bay station: None
2025-06-30 10:02:28,714 WARNING [make_plots] can't resolve a valid CHS station code for Boundary Bay
2025-06-30 10:02:36,814 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18541756 = 2025-06-30 03:30:00 UTC, 14.6% complete
2025-06-30 10:02:50,779 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb996/data for station 08074 Campbell River from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:03:59,468 WARNING [make_plots] invalid station number for Halfmoon Bay station: None
2025-06-30 10:03:59,469 WARNING [make_plots] can't resolve a valid CHS station code for Halfmoon Bay
2025-06-30 10:04:23,070 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb96d/data for station 07917 Nanaimo from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:04:49,827 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8562610 = 2025-06-30 04:06:00 UTC, 17.1% complete
2025-06-30 10:04:59,161 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 153, 35.4% complete
2025-06-30 10:05:07,431 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e13d0f4a073c4bbf93/data for station 07654 New Westminster from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:05:28,136 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd2d/data for station 07277 Patricia Bay from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:05:43,354 INFO [make_surface_current_tiles] starting PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/30jun25
2025-06-30 10:05:45,340 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb94c/data for station 07795 Point Atkinson from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:06:02,715 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e23d0f4a073c4bc060/data for station 08525 Port Renfrew from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:06:18,834 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1de3d0f4a073c4bb93c/data for station 07594 Sand Heads from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:06:35,931 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc43b/data for station 07786 Sandy Cove from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:06:52,860 WARNING [make_plots] invalid station number for Squamish station: None
2025-06-30 10:06:52,861 WARNING [make_plots] can't resolve a valid CHS station code for Squamish
2025-06-30 10:07:09,060 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1df3d0f4a073c4bbd1e/data for station 07120 Victoria from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:07:25,567 INFO [make_plots] retrieving obs water level data from https://api-iwls.dfo-mpo.gc.ca/api/v1/stations/5cebf1e43d0f4a073c4bc42d/data for station 07610 Woodwards Landing from 2025-06-25 00:00:00Z to 2025-07-02 11:55:00Z
2025-06-30 10:07:36,829 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18541931 = 2025-06-30 05:26:40 UTC, 22.7% complete
2025-06-30 10:07:43,059 INFO [make_plots] finished rendering nemo forecast publish 30jun25 figures
2025-06-30 10:07:43,061 INFO [make_plots] nemo publish plots for 2025-06-30 forecast completed
2025-06-30 10:07:43,069 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-30 10:07:43,242 INFO [manager] launching NextWorker(module='nowcast.workers.make_feeds', args=['forecast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 10:07:46,532 INFO [make_feeds] running in process 1242190
2025-06-30 10:07:46,533 INFO [make_feeds] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 10:07:46,535 INFO [make_feeds] publishing log messages to tcp://*:5565
2025-06-30 10:07:46,536 INFO [make_feeds] connected to skookum.eos.ubc.ca port 5555
2025-06-30 10:08:01,154 INFO [make_feeds] ATOM feeds for 2025-06-30 forecast run completed
2025-06-30 10:08:01,157 INFO [manager] checklist updated with [storm surge alert feeds] items from make_feeds worker
2025-06-30 10:09:50,306 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8562817 = 2025-06-30 06:24:00 UTC, 26.7% complete
2025-06-30 10:09:59,163 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 237, 54.9% complete
2025-06-30 10:10:21,693 INFO [make_surface_current_tiles] finished PDF concatenation and shrinking of tiles in /results/nowcast-sys/figures/surface_currents/forecast/30jun25
2025-06-30 10:10:21,719 INFO [make_surface_current_tiles] finished rendering of tiles for 2025-06-30 forecast into /results/nowcast-sys/figures/surface_currents/forecast/30jun25
2025-06-30 10:10:21,720 INFO [make_surface_current_tiles] surface current tile figures for 2025-06-30 forecast completed
2025-06-30 10:10:22,047 INFO [manager] checklist updated with [surface current tiles] items from make_surface_current_tiles worker
2025-06-30 10:12:36,814 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18542105 = 2025-06-30 07:22:40 UTC, 30.8% complete
2025-06-30 10:14:50,750 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8563025 = 2025-06-30 08:42:40 UTC, 36.3% complete
2025-06-30 10:14:59,163 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 324, 75.0% complete
2025-06-30 10:17:36,823 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18542280 = 2025-06-30 09:19:20 UTC, 38.9% complete
2025-06-30 10:19:51,194 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8563230 = 2025-06-30 10:59:20 UTC, 45.8% complete
2025-06-30 10:19:59,161 INFO [watch_ww3] nowcast on arbutus.cloud-nowcast: timestep: 408, 94.4% complete
2025-06-30 10:22:36,819 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18542454 = 2025-06-30 11:15:20 UTC, 46.9% complete
2025-06-30 10:24:51,685 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8563437 = 2025-06-30 13:17:20 UTC, 55.4% complete
2025-06-30 10:24:59,162 INFO [watch_ww3] nowcast WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-30 10:24:59,169 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-30 10:24:59,712 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'nowcast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 10:24:59,719 INFO [manager] launching NextWorker(module='nowcast.workers.run_ww3', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-30'], host='arbutus.cloud-nowcast')
2025-06-30 10:25:01,019 INFO [download_wwatch3_results] running in process 1243441
2025-06-30 10:25:01,020 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 10:25:01,021 INFO [download_wwatch3_results] publishing log messages to tcp://*:5564
2025-06-30 10:25:01,022 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-30 10:25:01,843 INFO [run_ww3] running in process 21976
2025-06-30 10:25:01,844 INFO [run_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 10:25:01,845 INFO [run_ww3] publishing log messages to tcp://*:5572
2025-06-30 10:25:01,846 INFO [run_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-30 10:25:01,850 INFO [run_ww3] Created run directory /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-06-30T102501.837143-0700
2025-06-30 10:25:01,851 INFO [run_ww3] forecast: launching /nemoShare/MEOPAR/nowcast-sys/wwatch3-runs/forecast_2025-06-30T102501.837143-0700/SoGWW3.sh on arbutus.cloud-nowcast
2025-06-30 10:25:01,859 INFO [run_ww3] forecast WaveWatch3 run for 2025-06-30 on arbutus.cloud-nowcast started
2025-06-30 10:25:01,872 INFO [manager] checklist updated with [WWATCH3 run] items from run_ww3 worker
2025-06-30 10:25:02,212 INFO [manager] launching NextWorker(module='nowcast.workers.watch_ww3', args=['arbutus.cloud-nowcast', 'forecast'], host='arbutus.cloud-nowcast')
2025-06-30 10:25:04,204 INFO [watch_ww3] running in process 22132
2025-06-30 10:25:04,205 INFO [watch_ww3] read config from /nemoShare/MEOPAR/nowcast-sys/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 10:25:04,207 INFO [watch_ww3] publishing log messages to tcp://*:5573
2025-06-30 10:25:04,208 INFO [watch_ww3] connected to skookum.eos.ubc.ca port 5555
2025-06-30 10:25:04,298 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-30 10:25:37,771 INFO [download_wwatch3_results] nowcast 2025-06-30 results files from arbutus.cloud-nowcast downloaded
2025-06-30 10:25:37,775 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-30 10:27:36,819 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18542628 = 2025-06-30 13:11:20 UTC, 55.0% complete
2025-06-30 10:29:52,162 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8563644 = 2025-06-30 15:35:20 UTC, 65.0% complete
2025-06-30 10:30:04,295 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 72, 11.1% complete
2025-06-30 10:32:36,820 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18542801 = 2025-06-30 15:06:40 UTC, 63.0% complete
2025-06-30 10:34:52,639 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8563851 = 2025-06-30 17:53:20 UTC, 74.6% complete
2025-06-30 10:35:04,297 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 147, 22.7% complete
2025-06-30 10:37:36,826 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18542975 = 2025-06-30 17:02:40 UTC, 71.1% complete
2025-06-30 10:39:53,123 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8564056 = 2025-06-30 20:10:00 UTC, 84.1% complete
2025-06-30 10:40:04,303 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 228, 35.2% complete
2025-06-30 10:42:36,817 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18543148 = 2025-06-30 18:58:00 UTC, 79.1% complete
2025-06-30 10:44:53,568 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8564263 = 2025-06-30 22:28:00 UTC, 93.7% complete
2025-06-30 10:45:04,307 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 315, 48.6% complete
2025-06-30 10:47:36,813 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18543321 = 2025-06-30 20:53:20 UTC, 87.1% complete
2025-06-30 10:49:53,998 INFO [watch_NEMO_agrif] 30jun25nowcast-agrif on orcinus-nowcast-agrif: timestep: 8564400 = 2025-06-30 23:59:20 UTC, 100.0% complete
2025-06-30 10:50:04,294 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 405, 62.5% complete
2025-06-30 10:52:36,815 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18543495 = 2025-06-30 22:49:20 UTC, 95.1% complete
2025-06-30 10:54:54,223 INFO [watch_NEMO_agrif] NEMO AGRIF run 11433312.orca2.ibb on orcinus-nowcast-agrif completed
2025-06-30 10:54:54,228 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO_agrif worker
2025-06-30 10:54:54,734 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['orcinus-nowcast-agrif', 'nowcast-agrif', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 10:54:56,090 INFO [download_results] running in process 1245231
2025-06-30 10:54:56,091 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 10:54:56,093 INFO [download_results] publishing log messages to tcp://*:5563
2025-06-30 10:54:56,094 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-30 10:54:56,095 INFO [download_results] downloading results from orcinus-nowcast-agrif:/global/scratch/dlatorne/nowcast-agrif/30jun25 to /results/SalishSea/nowcast-agrif.201702
2025-06-30 10:55:04,303 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 495, 76.4% complete
2025-06-30 10:55:17,647 INFO [download_results] nowcast-agrif 2025-06-30 results files from orcinus-nowcast-agrif downloaded
2025-06-30 10:55:17,654 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-30 10:55:17,983 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-agrif', 'research', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 10:55:21,014 INFO [make_plots] running in process 1245242
2025-06-30 10:55:21,015 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 10:55:21,016 INFO [make_plots] publishing log messages to tcp://*:5563
2025-06-30 10:55:21,017 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-30 10:55:21,314 INFO [make_plots] preparing render list for 2025-06-30 NEMO nowcast-agrif research figures
2025-06-30 10:55:21,315 INFO [make_plots] added baynes_sound_surface figure to 2025-06-30 NEMO nowcast-agrif research render list
2025-06-30 10:55:21,316 INFO [make_plots] starting to render nemo nowcast-agrif research 30jun25 figures
2025-06-30 10:55:31,720 INFO [make_plots] finished rendering nemo nowcast-agrif research 30jun25 figures
2025-06-30 10:55:31,723 INFO [make_plots] nemo research plots for 2025-06-30 nowcast-agrif completed
2025-06-30 10:55:31,727 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-30 10:57:36,817 INFO [watch_NEMO] nowcast-green on arbutus.cloud-nowcast: timestep: 18543600 = 2025-06-30 23:59:20 UTC, 100.0% complete
2025-06-30 11:00:04,312 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: timestep: 576, 88.9% complete
2025-06-30 11:02:36,832 INFO [watch_NEMO] nowcast-green NEMO run on arbutus.cloud-nowcast completed
2025-06-30 11:02:36,836 INFO [manager] checklist updated with [NEMO run] items from watch_NEMO worker
2025-06-30 11:02:37,354 INFO [manager] launching NextWorker(module='nowcast.workers.download_results', args=['arbutus.cloud-nowcast', 'nowcast-green', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 11:02:38,488 INFO [download_results] running in process 1245335
2025-06-30 11:02:38,489 INFO [download_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:02:38,490 INFO [download_results] publishing log messages to tcp://*:5563
2025-06-30 11:02:38,491 INFO [download_results] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:02:38,492 INFO [download_results] downloading results from arbutus.cloud-nowcast:/nemoShare/MEOPAR/SalishSea/nowcast-green/30jun25 to /results2/SalishSea/nowcast-green.202111
2025-06-30 11:05:04,300 INFO [watch_ww3] forecast on arbutus.cloud-nowcast: log.ww3 not found; continuing to watch...
2025-06-30 11:05:09,203 INFO [download_results] nowcast-green 2025-06-30 results files from arbutus.cloud-nowcast downloaded
2025-06-30 11:05:09,208 INFO [manager] checklist updated with [results files] items from download_results worker
2025-06-30 11:05:09,544 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['nemo', 'nowcast-green', 'research', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 11:05:09,549 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['nowcast-green'], host='localhost')
2025-06-30 11:05:09,554 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'chemistry', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 11:05:09,559 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'biology', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 11:05:09,564 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['day', 'physics', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 11:05:09,569 INFO [manager] launching NextWorker(module='nowcast.workers.archive_tarball', args=['nowcast-green', '2025-jun', 'robot.graham'], host='localhost')
2025-06-30 11:05:10,827 INFO [archive_tarball] running in process 1245385
2025-06-30 11:05:10,828 INFO [archive_tarball] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:05:10,829 INFO [archive_tarball] publishing log messages to tcp://*:5563
2025-06-30 11:05:10,829 INFO [archive_tarball] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:05:10,830 INFO [archive_tarball] creating /ocean/dlatorne/nowcast-green.202111-jun25.tar from /results2/SalishSea/nowcast-green.202111/*jun25/
2025-06-30 11:05:11,005 INFO [ping_erddap] running in process 1245381
2025-06-30 11:05:11,007 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:05:11,007 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-06-30 11:05:11,008 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:05:11,014 INFO [ping_erddap] nowcast-green ERDDAP dataset flag file(s) created
2025-06-30 11:05:11,015 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-30 11:05:15,404 INFO [make_averaged_dataset] running in process 1245383
2025-06-30 11:05:15,413 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:05:15,427 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-06-30 11:05:15,440 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:05:15,449 INFO [make_averaged_dataset] creating day-averaged dataset for 30-Jun-2025 biology
2025-06-30 11:05:15,462 INFO [make_averaged_dataset] running in process 1245382
2025-06-30 11:05:15,476 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:05:15,485 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-06-30 11:05:15,498 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:05:15,511 INFO [make_averaged_dataset] creating day-averaged dataset for 30-Jun-2025 chemistry
2025-06-30 11:05:15,534 INFO [make_averaged_dataset] running in process 1245384
2025-06-30 11:05:15,547 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:05:15,556 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2025-06-30 11:05:15,569 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:05:15,583 INFO [make_averaged_dataset] creating day-averaged dataset for 30-Jun-2025 physics
2025-06-30 11:05:19,871 INFO [make_plots] running in process 1245379
2025-06-30 11:05:19,872 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:05:19,886 INFO [make_plots] publishing log messages to tcp://*:5567
2025-06-30 11:05:19,900 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:05:20,436 INFO [make_plots] preparing render list for 2025-06-30 NEMO nowcast-green research figures
2025-06-30 11:05:22,762 INFO [make_plots] added temperature_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:23,618 INFO [make_plots] added nitrate_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:24,527 INFO [make_plots] added ammonium_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:25,400 INFO [make_plots] added silicon_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:26,296 INFO [make_plots] added dissolved_organic_nitrogen_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:27,283 INFO [make_plots] added particulate_organic_nitrogen_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:28,118 INFO [make_plots] added biogenic_silicon_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:29,063 INFO [make_plots] added diatoms_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:30,021 INFO [make_plots] added flagellates_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:30,976 INFO [make_plots] added microzooplankton_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:31,911 INFO [make_plots] added mesozooplankton_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:32,726 INFO [make_plots] added turbidity_thalweg_and_surface figures to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:34,197 INFO [make_plots] added temperature_salinity_timeseries figure to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:34,213 INFO [make_plots] added nitrate_diatoms_timeseries figure to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:34,226 INFO [make_plots] added diatoms_flagellates_timeseries figure to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:34,235 INFO [make_plots] added z1_z2_zooplankton_timeseries figure to 2025-06-30 NEMO nowcast-green research render list
2025-06-30 11:05:34,249 INFO [make_plots] starting to render nemo nowcast-green research 30jun25 figures
2025-06-30 11:05:50,274 INFO [make_averaged_dataset] day-averaged dataset for 30-Jun-2025 physics created
2025-06-30 11:05:50,277 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-30 11:05:50,591 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['month', 'physics', '--run-date', '2025-06-01'], host='localhost')
2025-06-30 11:05:53,376 INFO [make_averaged_dataset] running in process 1245476
2025-06-30 11:05:53,379 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:05:53,381 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2025-06-30 11:05:53,382 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:05:53,383 INFO [make_averaged_dataset] creating month-averaged dataset for Jun-2025 physics
2025-06-30 11:06:23,717 INFO [make_averaged_dataset] day-averaged dataset for 30-Jun-2025 chemistry created
2025-06-30 11:06:23,720 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-30 11:06:24,078 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['month', 'chemistry', '--run-date', '2025-06-01'], host='localhost')
2025-06-30 11:06:30,344 INFO [make_averaged_dataset] running in process 1245513
2025-06-30 11:06:30,344 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:06:30,345 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-06-30 11:06:30,346 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:06:30,347 INFO [make_averaged_dataset] creating month-averaged dataset for Jun-2025 chemistry
2025-06-30 11:06:44,808 INFO [make_averaged_dataset] day-averaged dataset for 30-Jun-2025 biology created
2025-06-30 11:06:44,812 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-30 11:06:45,168 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['month', 'biology', '--run-date', '2025-06-01'], host='localhost')
2025-06-30 11:06:50,234 INFO [make_averaged_dataset] running in process 1245541
2025-06-30 11:06:50,236 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:06:50,237 INFO [make_averaged_dataset] publishing log messages to tcp://*:5564
2025-06-30 11:06:50,238 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:06:50,239 INFO [make_averaged_dataset] creating month-averaged dataset for Jun-2025 biology
2025-06-30 11:09:32,648 INFO [make_averaged_dataset] month-averaged dataset for Jun-2025 physics created
2025-06-30 11:09:32,652 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-30 11:09:33,018 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['month', 'grazing', '--run-date', '2025-06-01'], host='localhost')
2025-06-30 11:09:45,326 INFO [make_averaged_dataset] running in process 1245603
2025-06-30 11:09:45,329 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:09:45,342 INFO [make_averaged_dataset] publishing log messages to tcp://*:5566
2025-06-30 11:09:45,343 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:09:45,344 INFO [make_averaged_dataset] creating month-averaged dataset for Jun-2025 grazing
2025-06-30 11:09:59,450 INFO [make_averaged_dataset] month-averaged dataset for Jun-2025 chemistry created
2025-06-30 11:09:59,454 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-30 11:10:04,309 INFO [manager] checklist updated with [WWATCH3 run] items from watch_ww3 worker
2025-06-30 11:10:04,313 INFO [watch_ww3] forecast WWATCH3 run on arbutus.cloud-nowcast completed
2025-06-30 11:10:04,835 INFO [manager] launching NextWorker(module='nowcast.workers.download_wwatch3_results', args=['arbutus.cloud-nowcast', 'forecast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 11:10:06,015 INFO [download_wwatch3_results] running in process 1245626
2025-06-30 11:10:06,016 INFO [download_wwatch3_results] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:10:06,017 INFO [download_wwatch3_results] publishing log messages to tcp://*:5565
2025-06-30 11:10:06,018 INFO [download_wwatch3_results] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:10:51,927 INFO [make_averaged_dataset] month-averaged dataset for Jun-2025 biology created
2025-06-30 11:10:51,930 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-30 11:10:52,627 INFO [download_wwatch3_results] forecast 2025-06-30 results files from arbutus.cloud-nowcast downloaded
2025-06-30 11:10:52,631 INFO [manager] checklist updated with [WWATCH3 results files] items from download_wwatch3_results worker
2025-06-30 11:10:52,987 INFO [manager] launching NextWorker(module='nowcast.workers.update_forecast_datasets', args=['wwatch3', 'forecast', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 11:10:59,453 INFO [update_forecast_datasets] running in process 1245648
2025-06-30 11:10:59,469 INFO [update_forecast_datasets] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:10:59,483 INFO [update_forecast_datasets] publishing log messages to tcp://*:5564
2025-06-30 11:10:59,495 INFO [update_forecast_datasets] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:10:59,504 INFO [update_forecast_datasets] updating wwatch3 most_recent_forecast directory from forecast/30jun25 run
2025-06-30 11:10:59,526 INFO [update_forecast_datasets] updated wwatch3 most_recent_forecast directory from forecast/30jun25 run
2025-06-30 11:10:59,530 INFO [update_forecast_datasets] updating wwatch3 forecast directory for forecast/30jun25 run
2025-06-30 11:10:59,681 INFO [update_forecast_datasets] updated wwatch3 forecast directory for forecast/30jun25 run: /results/SalishSea/rolling-forecasts/wwatch3
2025-06-30 11:10:59,688 INFO [update_forecast_datasets] wwatch3 2025-06-30 forecast rolling forecast datasets updated
2025-06-30 11:10:59,715 INFO [manager] checklist updated with [update forecast datasets] items from update_forecast_datasets worker
2025-06-30 11:11:00,003 INFO [manager] launching NextWorker(module='nowcast.workers.ping_erddap', args=['wwatch3-forecast'], host='localhost')
2025-06-30 11:11:01,763 INFO [ping_erddap] running in process 1245654
2025-06-30 11:11:01,764 INFO [ping_erddap] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:11:01,765 INFO [ping_erddap] publishing log messages to tcp://*:5564
2025-06-30 11:11:01,766 INFO [ping_erddap] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:11:01,768 INFO [ping_erddap] wwatch3-forecast ERDDAP dataset flag file(s) created
2025-06-30 11:11:01,770 INFO [manager] checklist updated with [ERDDAP flag files] items from ping_erddap worker
2025-06-30 11:11:02,104 INFO [manager] launching NextWorker(module='nowcast.workers.make_plots', args=['wwatch3', 'forecast', 'publish', '--run-date', '2025-06-30'], host='localhost')
2025-06-30 11:11:06,421 INFO [make_plots] running in process 1245659
2025-06-30 11:11:06,422 INFO [make_plots] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:11:06,433 INFO [make_plots] publishing log messages to tcp://*:5564
2025-06-30 11:11:06,441 INFO [make_plots] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:11:06,450 INFO [make_plots] preparing render list for 2025-06-30 WaveWatch3 forecast publish figures
2025-06-30 11:11:06,455 INFO [make_plots] added Halibut Bank figure to 2025-06-30 WaveWatch3 forecast publish render list
2025-06-30 11:11:06,466 INFO [make_plots] added Sentry Shoal figure to 2025-06-30 WaveWatch3 forecast publish render list
2025-06-30 11:11:06,476 INFO [make_plots] starting to render wwatch3 forecast publish 30jun25 figures
2025-06-30 11:11:08,112 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46146 Halibut Bank
2025-06-30 11:13:23,651 INFO [make_averaged_dataset] month-averaged dataset for Jun-2025 grazing created
2025-06-30 11:13:23,655 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-30 11:13:24,004 INFO [manager] launching NextWorker(module='nowcast.workers.make_averaged_dataset', args=['month', 'growth', '--run-date', '2025-06-01'], host='localhost')
2025-06-30 11:13:28,846 INFO [make_averaged_dataset] running in process 1245701
2025-06-30 11:13:28,847 INFO [make_averaged_dataset] read config from /SalishSeaCast/SalishSeaNowcast/config/nowcast.yaml
2025-06-30 11:13:28,849 INFO [make_averaged_dataset] publishing log messages to tcp://*:5565
2025-06-30 11:13:28,850 INFO [make_averaged_dataset] connected to skookum.eos.ubc.ca port 5555
2025-06-30 11:13:28,852 INFO [make_averaged_dataset] creating month-averaged dataset for Jun-2025 growth
2025-06-30 11:14:41,718 INFO [make_averaged_dataset] month-averaged dataset for Jun-2025 growth created
2025-06-30 11:14:41,770 INFO [manager] checklist updated with [averaged dataset] items from make_averaged_dataset worker
2025-06-30 11:21:51,402 INFO [make_plots] retrieving available real-time buoy data from https://www.ndbc.noaa.gov/data/realtime2/ for buoy 46131 Sentry Shoal
2025-06-30 11:22:00,225 INFO [make_plots] finished rendering wwatch3 forecast publish 30jun25 figures
2025-06-30 11:22:00,229 INFO [make_plots] wwatch3 publish plots for 2025-06-30 forecast completed
2025-06-30 11:22:00,256 INFO [manager] checklist updated with [plots] items from make_plots worker
2025-06-30 12:01:56,436 INFO [archive_tarball] creating /ocean/dlatorne/nowcast-green.202111-jun25.index from /ocean/dlatorne/nowcast-green.202111-jun25.tar
2025-06-30 12:02:31,696 INFO [archive_tarball] rsync-ing /ocean/dlatorne/nowcast-green.202111-jun25.tar and index to robot.graham:/nearline/rrg-allen/SalishSea/nowcast-green.202111/
2025-06-30 12:14:26,327 INFO [make_plots] finished rendering nemo nowcast-green research 30jun25 figures
2025-06-30 12:14:26,338 INFO [make_plots] nemo research plots for 2025-06-30 nowcast-green completed
2025-06-30 12:14:26,399 INFO [manager] checklist updated with [plots] items from make_plots worker